Skip to main content
New
Stack Overflow Jobs powered by Indeed: A job site that puts thousands of tech jobs at your fingertips (U.S. only). Search jobs
9 votes

Is there a Git for data?

Another open-source one that has popped up is DVC https://github.com/iterative/dvc Data Science Version Control or DVC is an open-source tool for data science and machine learning projects. With a ...
philshem's user avatar
  • 17.6k
7 votes
Accepted

What kind of spreadsheet/application are the Socrata datasets built on?

Socrata actually isn't built on top of an existing spreadsheet application or platform. We've built our own platform using a number of different open source technologies. Customer data is stored in ...
chrismetcalf's user avatar
7 votes

Wikipedia table to JSON (or other machine-readable format)

The wikitextparser Python module has good support for converting tables into structured formats. Example: import wikitextparser as wtp import requests import json r = requests.get('http://en....
maxschlepzig's user avatar
5 votes

Is there a Git for data?

Yes there is one, it is called noms https://github.com/attic-labs/noms Noms is a decentralized database philosophically descendant from the Git version control system.
JohnDoe's user avatar
  • 151
4 votes

How can I work with a 4GB csv file?

I recently had to parse the 6GB NPPES file and here is how I did it: $ wget http://download.cms.gov/nppes/NPPES_Data_Dissemination_July_2017.zip $ unzip NPPES_Data_Dissemination_July_2017.zip $ split ...
Andrei Sura's user avatar
4 votes

How can I work with a 4GB csv file?

I am a big fan of tad which crunches these kind of files easily.
Steven's user avatar
  • 141
4 votes

How to add a claim to an existing Wikidata item, from command line?

You can use wikidata-agent for that: curl -XPOST http://your.wikidata.agent.instance/claim -d 'entity=Q5369879&property=P137&value=Q39' [Update]: you can now actually add claims using ...
maxlath's user avatar
  • 266
3 votes

Tool to extract the main concepts/topics from web pages

Thanks Tasos for your very helpful DBpedia Spotlight Python code sample above. However, converting it from Python 2 to Python 3+ wasn't entirely straightforward so thought I'd post the revised version....
neilb's user avatar
  • 31
3 votes

Whats the simplest way to get 1000 twitter handles of users in USA who are less than 21 years of age?

As mentioned above, Twitter has no searchable age category, but there are some clever ways to deduce Filter on first names that barely existed more than 21 years ago. For example, Harper. You can ...
philshem's user avatar
  • 17.6k
3 votes

Whats the simplest way to get 1000 twitter handles of users in USA who are less than 21 years of age?

I don't think there is a specific method for what you are asking for. There are some methods of acquiring a large number of handles, although they each come with downsides. I typically use the twitteR ...
Oliver Frost's user avatar
3 votes

Wikipedia table to JSON (or other machine-readable format)

Sometimes it pays off to look one step ahead, i.e. to check if the items and its properties represented by a Wikipedia table are also available in the Wikidata knowledge base. Depending on the data, ...
maxschlepzig's user avatar
3 votes

Good tools to parse repetitive unstructured data

Take a look at UTAH: https://github.com/sonalake/utah-parser It's a good tool for handling files like this.
Daniel Bray's user avatar
2 votes

Is there a Git for data?

Recently (november 2016), ClusterHQ released a new tool called fli that seem to be promising. I have not tried it yet, but it seems it plays well with many database system (mysql, mongodb, elastic, ...
jvtrudel's user avatar
  • 121
2 votes

Is there a Git for data?

For RDF-data: Quit Semantic Web and linked data are made by RDF... There are a solution at http://aksw.org/Projects/Quit.html Any kind of RDF: XML, JSON-LD, HTML-RDFa, etc. For CSV data: ...
Peter Krauss's user avatar
2 votes

Is there a Git for data?

qri (pronounced "query") Qri is a distributed dataset version control system built with peer-2-peer data exchange. Peers create datasets, which are stored in versions. Qri peers form a distributed ...
philshem's user avatar
  • 17.6k
2 votes

Where to upload media not yet compatible with Wikimedia Commons licenses? ("open dark archive")

The website should keep the uploaded images, and provide visitors with as much as what the copyright laws allow (for instance only metadata and a fair-use thumbnail) Might be someone would want to ...
user930067's user avatar
2 votes

Sentiment search engine

I have actually been working on something very similar; a search engine for restaurants in London. The search engine crawls popular social media platforms (TripAdvisor, Open Table etc.) on a daily ...
Pritz's user avatar
  • 21
2 votes

How can I work with a 4GB csv file?

You can try nitroproc. It's currently a beta version I think, but allows to sort/summarize/filter/etc. It's free.
bob's user avatar
  • 21
2 votes

Long-term search trends by domain

You could try this open source library in node: https://www.npmjs.com/package/google-trends-api It seems to have several Google trends functions available.
NickZ's user avatar
  • 21
2 votes

If Wordpress is an appropriate tool for loading open data into a system's front end, what would I use for the backend?

your IT professional was right. You will need to have an FTP account that your developers can access and change where needed. You are able to set up FTP accounts through your ISP (internet service ...
user13072's user avatar
2 votes
Accepted

If Wordpress is an appropriate tool for loading open data into a system's front end, what would I use for the backend?

Wordpress is the backend of Wordpress sites. You can use an FTP connection, but you do not have to have one. I run WordPress blog, and various installations for clients/development on my server, which ...
albert's user avatar
  • 11.9k
2 votes

Data integration tool that supports Semantic Web

I would suggest looking at https://data.world for the data aggregation right now (disclaimer, I work there...but it's a free resource). You can aggregate multiple data sources quite easily. Our entire ...
scuttlemonkey's user avatar
1 vote

Scientometric/bibliometric data retrieval from a list of DOI

Other than CrossRef alone, OpenCitations might be of help to you as well.
anpami's user avatar
  • 307
1 vote

Open data set to calculate the correlation between different brands

You should look for semantic similarity. The normalized Google distance should be a good place to start, and gives the kind of [0-1] rating you are looking for. The Normalized Google Distance is a ...
Ho1's user avatar
  • 188
1 vote

How can I work with a 4GB csv file?

I've had good luck in R (without resorting to fancy packages) of reading CSVs up to and beyond that size. The nice thing about getting it in RAM, if you can swing it, is very fast manipulation.
Mike Dolan Fliss's user avatar
1 vote

Is there a tool to convert geoRSS into geoJSON?

Yes, ogr2ogr does this job efficiently. Install, GDAL/OGR then run below command. ogr2ogr -f GeoJSON target.geojson geoRSS.xml (here target.geojson is the geojson file to be generated, while geoRSS....
Jadav Bheda's user avatar
1 vote

Stemming of long English text

You can also check out the algorithm used on https://www.word-grabber.com/words-with-friends-cheat that finds word possibilities from letters.
Stevie Ray's user avatar
1 vote

Twitter crawlers for tweets, retweets and social network

With www.followthehashtag.com twitter analytics service you will be able to get Twitter data for free up to 9 days back or track some thousand tweets or, in premium version since 2006, with no total ...
Enrique R Rivera's user avatar
1 vote

Wikipedia table to JSON (or other machine-readable format)

TableTools2 https://addons.mozilla.org/en-US/firefox/addon/tabletools2/ It's a Firefox addon that enables right-click copying of the table like this: It works quite well and is very stable.
Anton Tarasenko's user avatar
1 vote

Is there a Git for data?

https://www.liquidata.co is supposed to be "git for data".
Make42's user avatar
  • 151

Only top scored, non community-wiki answers of a minimum length are eligible