9 votes

Is there a Git for data?

Another open-source one that has popped up is DVC https://github.com/iterative/dvc Data Science Version Control or DVC is an open-source tool for data science and machine learning projects. With a ...
  • 17.4k
7 votes

Wikipedia table to JSON (or other machine-readable format)

The wikitextparser Python module has good support for converting tables into structured formats. Example: import wikitextparser as wtp import requests import json r = requests.get('http://en....
7 votes
Accepted

Scientometric/bibliometric data retrieval from a list of DOI

Maybe the rcrossref package for R is helpful. To find the number of citations, You can do things such as library(rcrossref) # returns list: # pap <- cr_works(dois = "10.1371/journal.pone....
  • 1,184
7 votes
Accepted

What kind of spreadsheet/application are the Socrata datasets built on?

Socrata actually isn't built on top of an existing spreadsheet application or platform. We've built our own platform using a number of different open source technologies. Customer data is stored in ...
6 votes
Accepted

How do I share Open Data with others on this SE site?

We have recently created the OpenData StackExchange organization on datahub.io. Welcome to the Datahub, the free, powerful data management platform from the Open Knowledge Foundation, based on the ...
  • 17.4k
5 votes

Is there a Git for data?

Yes there is one, it is called noms https://github.com/attic-labs/noms Noms is a decentralized database philosophically descendant from the Git version control system.
  • 151
5 votes

How can I work with a 4GB csv file?

If you're on Windows, I can't sing the praises of LogParser high enough. It allows you to query files in a wide variety of formats (mostly log formats as that's what it was meant for, but XML and CSV ...
5 votes

Sentiment search engine

The first source of raw interactions that fit your needs that comes to mind is Twitter. NCSU's Tweet Sentiment Visualization http://www.csc.ncsu.edu/faculty/healey/tweet_viz/tweet_app/ seems pretty ...
5 votes

Time-Series data viewer

It's hard to suggest a good tool without knowing how deep into programming you want to go, or if the tool is for exploration or presentation. But here is a sample of many good tools out there: ...
4 votes

How can I work with a 4GB csv file?

I recently had to parse the 6GB NPPES file and here is how I did it: $ wget http://download.cms.gov/nppes/NPPES_Data_Dissemination_July_2017.zip $ unzip NPPES_Data_Dissemination_July_2017.zip $ split ...
4 votes

How can I work with a 4GB csv file?

I am a big fan of tad which crunches these kind of files easily.
  • 141
4 votes
Accepted

Retrieve affiliations from research paper

Many of the bibliographic databases offer APIs, but they might not be 100% open. Typically, you can't get access to the ones that charge for access unless your institution has a subscription to the ...
  • 4,495
4 votes

What forums / boards do you use to answer open data questions?

This one obviously. I also use http://reddit.com/r/opendata, and more recently the Open Knowledge Foundation's forums (http://discuss.okfn.org/) but that's for a project specific to them so I can't ...
  • 323
4 votes

Cleaning up addresses in a large data set

Check this out: https://parserator.datamade.us/ source code here: https://github.com/datamade/usaddress
  • 301
4 votes

How to add a claim to an existing Wikidata item, from command line?

You can use wikidata-agent for that: curl -XPOST http://your.wikidata.agent.instance/claim -d 'entity=Q5369879&property=P137&value=Q39' [Update]: you can now actually add claims using ...
  • 256
3 votes

How might I go about visualising historical temperature CSV data?

There has been a bunch of statistical work in this area (spatio-temporal modeling and extreme value theory); there have been several programs at SAMSI that have generated good peer reviewed papers ( ...
3 votes

Tool to extract the main concepts/topics from web pages

Thanks Tasos for your very helpful DBpedia Spotlight Python code sample above. However, converting it from Python 2 to Python 3+ wasn't entirely straightforward so thought I'd post the revised version....
  • 31
3 votes

Wikipedia table to JSON (or other machine-readable format)

Sometimes it pays off to look one step ahead, i.e. to check if the items and its properties represented by a Wikipedia table are also available in the Wikidata knowledge base. Depending on the data, ...
3 votes

Good tools to parse repetitive unstructured data

Take a look at UTAH: https://github.com/sonalake/utah-parser It's a good tool for handling files like this.
3 votes
Accepted

Looking for web scraper that works on YouTube channel video catalogs

TubeKit might be of interest to you: TubeKit is a toolkit for creating YouTube crawlers. It allows one to build one's own crawler that can crawl YouTube based on a set of seed queries and collect ...
3 votes

Looking for web scraper that works on YouTube channel video catalogs

On Linux use youtube-dl command to download videos straight off of YouTube. To install youtube-dl run sudo pip install youtube-dl To download an entire channel: youtube-dl -citw ytuser:<USER> ...
  • 8,142
3 votes

What forums / boards do you use to answer open data questions?

National Institute for Computer-Assisted Reporting, mailing list NICAR is an oldie and a goodie. http://www.ire.org/resource-center/listservs/archives/
  • 5,108
3 votes
Accepted

Mapping CSV Header to RDF + SPARQL console

Before building a completely new system from scratch, you should first check if an existing system satisfies your requirements. There are a few possible candidates out there: As outlined in this ...
3 votes

Cleaning up addresses in a large data set

Standard (basic) USA address decomposition is: Street Secondary Address Unit (e.g., floor, bldg.) City State Postal Code Secondary Postal Code (e.g., the 1234 part of 97123-1234) More detailed ...
3 votes

Creating data from web tables with import.io failed - other tools?

Data Scraping Studio can do this and it's free software. I just created a scraping agent using this tool and extracted all tables into one using a REGEX parser. The extracted data should be exactly ...
3 votes

Whats the simplest way to get 1000 twitter handles of users in USA who are less than 21 years of age?

I don't think there is a specific method for what you are asking for. There are some methods of acquiring a large number of handles, although they each come with downsides. I typically use the twitteR ...
3 votes

Whats the simplest way to get 1000 twitter handles of users in USA who are less than 21 years of age?

As mentioned above, Twitter has no searchable age category, but there are some clever ways to deduce Filter on first names that barely existed more than 21 years ago. For example, Harper. You can ...
  • 17.4k
2 votes

Is there a Git for data?

Recently (november 2016), ClusterHQ released a new tool called fli that seem to be promising. I have not tried it yet, but it seems it plays well with many database system (mysql, mongodb, elastic, ...
  • 121
2 votes

Is there a Git for data?

For RDF-data: Quit Semantic Web and linked data are made by RDF... There are a solution at http://aksw.org/Projects/Quit.html Any kind of RDF: XML, JSON-LD, HTML-RDFa, etc. For CSV data: ...

Only top scored, non community-wiki answers of a minimum length are eligible