9 votes

Is there a Git for data?

Since this question was originally asked, the Dat project has made a lot of progress. Originally conceived by Max Ogden, it now has several other developers working on it. dat is an open source ...
user avatar
  • 5,468
8 votes

Is there a Git for data?

Another open-source one that has popped up is DVC https://github.com/iterative/dvc Data Science Version Control or DVC is an open-source tool for data science and machine learning projects. With a ...
user avatar
  • 17.1k
8 votes

How can I work with a 4GB csv file?

On Windows, there is also a software called Delimit ("Open data files up to 2 billion rows and 2 million columns large!") http://delimitware.com For instance it can split, sort and extract only some ...
user avatar
  • 181
7 votes

Wikipedia table to JSON (or other machine-readable format)

The wikitextparser Python module has good support for converting tables into structured formats. Example: import wikitextparser as wtp import requests import json r = requests.get('http://en....
user avatar
7 votes
Accepted

Scientometric/bibliometric data retrieval from a list of DOI

Maybe the rcrossref package for R is helpful. To find the number of citations, You can do things such as library(rcrossref) # returns list: # pap <- cr_works(dois = "10.1371/journal.pone....
user avatar
  • 1,164
7 votes
Accepted

What kind of spreadsheet/application are the Socrata datasets built on?

Socrata actually isn't built on top of an existing spreadsheet application or platform. We've built our own platform using a number of different open source technologies. Customer data is stored in ...
user avatar
6 votes

Graph visualization/analysis tool?

If your main focus is to visualize the data and have graph metrics I would recommend this list. If you want an off the shelf package Gephi - Desktop application, Open Source license Cytoscape - ...
user avatar
  • 161
6 votes

How can I work with a 4GB csv file?

This answer is not really useful for non-programmers, but if could manage some programming in perl, the Parse::CSV module is especially designed for this task. From the doc: It provides a flexible ...
user avatar
  • 235
6 votes
Accepted

How do I share Open Data with others on this SE site?

We have recently created the OpenData StackExchange organization on datahub.io. Welcome to the Datahub, the free, powerful data management platform from the Open Knowledge Foundation, based on the ...
user avatar
  • 17.1k
6 votes

Collect a list of open data systems

I've compiled a fairly extensive list of open government portals both in US and around the world in the last few days. I am working on code to support crowdsourcing the catalog. I also have a CSV ...
user avatar
6 votes

Converting Geographic Coordinates to New York City neighboorhood names

tamu has a killer geocoder, as well as a definitive list of other free geocoders. all you have to do is upload your data http://geoservices.tamu.edu/Services/Geocode/BatchProcess/ http://geoservices....
user avatar
  • 11.8k
5 votes

Is there a Git for data?

Yes there is one, it is called noms https://github.com/attic-labs/noms Noms is a decentralized database philosophically descendant from the Git version control system.
user avatar
  • 151
5 votes

Graph visualization/analysis tool?

My team builds https://vida.io a tool for creating data visualization templates. We support d3.js templates. You can see a lot of examples on our site: https://vida.io/explore For simple chart/...
user avatar
  • 151
5 votes

How can I work with a 4GB csv file?

Well, in short, Talend Open Studio for Data Integration is an ETL. It can be used for many use cases, including data migration, files processing, etc. You can easily build jobs using a visual editor ...
user avatar
  • 171
5 votes

How can I work with a 4GB csv file?

If you're on Windows, I can't sing the praises of LogParser high enough. It allows you to query files in a wide variety of formats (mostly log formats as that's what it was meant for, but XML and CSV ...
user avatar
5 votes

Converting Geographic Coordinates to New York City neighboorhood names

Zillow provides Shapefiles for neighborhoods for the largest US Cities (includes NYC) for free (CC license). You can find them here: http://www.zillow.com/howto/api/neighborhood-boundaries.htm
user avatar
5 votes

Best way to convert Excel Files to Open Data Formats

I don't believe that you will find a "ready" library that will be the case for all your excel files. My advice is to create a script by yourself in base of your file structures. Especially if the ...
user avatar
  • 4,716
5 votes

Best way to convert Excel Files to Open Data Formats

For example with Python, using Pandas: import pandas as pd data = pd.from_excel("path_to.xls", sheetname="sheet1") data.to_csv("path_to.csv", encoding="utf8") (As simple as that.) Pandas can read ...
user avatar
5 votes

Sentiment search engine

The first source of raw interactions that fit your needs that comes to mind is Twitter. NCSU's Tweet Sentiment Visualization http://www.csc.ncsu.edu/faculty/healey/tweet_viz/tweet_app/ seems pretty ...
user avatar
5 votes

Time-Series data viewer

It's hard to suggest a good tool without knowing how deep into programming you want to go, or if the tool is for exploration or presentation. But here is a sample of many good tools out there: ...
4 votes

Graph visualization/analysis tool?

Quadrigram (www.quadrigram.com) has good graph visualizations in both 2D and 3D. They are relatively easy to set and publish. You can also combine them with Maps and other visualizations Checkout ...
user avatar
4 votes

Is there a Git for data?

Git's diff-ing and merge-ing functionality can be extended using custom drivers. I just wrote a custom merging driver for json files that merges based on location in the tree instead of on a line-by-...
user avatar
4 votes

How can I work with a 4GB csv file?

I recently had to parse the 6GB NPPES file and here is how I did it: $ wget http://download.cms.gov/nppes/NPPES_Data_Dissemination_July_2017.zip $ unzip NPPES_Data_Dissemination_July_2017.zip $ split ...
user avatar
4 votes

How can I work with a 4GB csv file?

I am a big fan of tad which crunches these kind of files easily.
user avatar
  • 141
4 votes

Collect a list of open data systems

There's an epic list of text-processing and text-mining tools at the Bamboo DiRT wiki. Go to town!
user avatar
  • 526
4 votes

Do you know a technique for text mining judicial decisions?

If you are interested to just scrape some data from web pages, then like @magdmartin wrote, you just need to write some code to download the HTML (python requests package) and then to parse the HTML (...
user avatar
  • 17.1k
4 votes

What is the best tool for doing explorative analysis of databases?

looks like possibly same question on stackoverflow: https://stackoverflow.com/questions/12835572/how-to-use-pure-sql-for-exploratory-data-analysis most results i found recommend using R http://www....
user avatar
  • 11.8k
4 votes

Wiki site for data tables

A hopeful candidate is Dat, which aspires to become what Git (and GitHub) is for code. With this in mind, searching for "git for data" yields a lot of interesting results. In that context, the article ...
user avatar
  • 2,784
4 votes

Best way to convert Excel Files to Open Data Formats

You may want to look at this previous stackoverflow question on converting Excel to CSV in batch mode on Linux. Once it is in CSV, you have lots of options of converting to other formats - but CSV is ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible