62

This is the kind of thing that the csvkit was built for: csvgrep -c "Healthcare Provider Taxonomy Code_1" -r '^282N' npidata_20050523-20131110.csv > hospitals.csv csvkit is a suite of utilities for converting to and working with CSV, the king of tabular file formats. A little more efficiently, you could do: zcat NPPES_Data_Dissemination_Nov_2013....


42

Basic answer afaik is: "No". As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures. Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as ...


40

I recently stumbled on this article by the Open Knowledge Foundation regarding the design of a graphical interface to diff tabular data called daff. It can also be tested and forked on GitHub.


31

There is a project dat that addresses what you are looking for. It is now in beta, featuring a command line tool with similar design like git. There are also APIs for Javascript (dat-core), Python (datpy) and R (rdat). Supported input data formats for tabular data are (as of August 2015): CSV, TSV, JSON, and newline delimited JSON. It also supports ...


26

On Windows, SweetScape 010 Editor is the best application I am aware of to open/edit large files (easily up to 25 GB). It took around 10 seconds on my computer to open your 4 GB file (SSD): More such tools: Text editor to open big (giant, huge, large) text files


20

Please note that git has two configuration commands: git config filter.<driver>.clean git config filter.<driver>.smudge The clean filter can be used e.g. to sort all records in a CSV file except the top row, thus making re-ordering irrelevant. Quoting gitattributes(5) : A filter driver consists of a clean command and a smudge command, either of ...


17

As you're only taking a portion of the file, you may be able to use simple tools to subset it before processing. That may get it down to a reasonable size to work with. If you're working on a posix (ie, unix-like) system, you can use shell commands to reduce the file: zcat -cfilename| grep(pattern to match hospitals only)>outputFile This lets you ...


15

Can you provide a more comprehensive list of the pain points you're experiencing with Git? Git works great for data. In addition to the config flag Deer Hunter mentioned for sorting, you can also teach git to diff on a word-by-word (rather than line-by-line basis), by simply passing the --word-diff flag.


14

you can connect to the file with sql and run your analysis from there. i have written extremely detailed r code (r is free and open source) about how to work with the nppes from your laptop here: http://asdfree.com/national-plan-and-provider-enumeration-system-nppes.html if you have never used r before, check out http://twotorials.com for a crash course ...


14

Data Here is my take on it: I use R and its IDE RStudio. The hard part, cleaning the data, is luckily done. Sharing the CSV via a dropbox link is not bad. The file is well structured. To improve it you could add a licence and provide a bit more information about the source. For more information see our certificates. If you want to publish in a more "...


13

If you already have a nicely formatted CSV then why not publish that? You can publish in both formats if you really want to do the RDFa too.


12

You can use Google Spreadsheets ImportHTML formula as detailed in this Liberating HTML Tables (using Google Spreadsheet) tutorial on School of Data by Tony Hirst - it includes a specific walk-through for Wikipedia. The essence is to do: =importHTML("","table",N) In your case you could try: =importHTML("http://en.wikipedia.org/wiki/...


11

There is no simple answer to this question, because ZIP codes do not represent geographical areas. They represent postal delivery routes, which are sometimes simply a bank of PO boxes in a specific post office, and are sometimes an organization like a University which has its own internal mail processing services. Therefore, not all ZIP Codes can truly be ...


11

Others have mentioned way to pull apart this file incrementally. It seems to me like you are also commenting on use of resources for a large file. For some solutions you can incrementally read the compressed file uncompressing as you go and feed it through the csv module. For example in python with gzip'd input you would do this by: import csv import ...


10

There has been some mention about this being a complex problem. It's not hard at all. In fact, matching a zipcode to the corresponding cities is super simple. Granted, trying to find an accurate shapefile that corresponds to a given zipcode, that is a more challenging issue. But simply matching zipcode to city/cities... very easy. The USPS data has a ...


10

There are streaming CSV parsers, that only look at a small window of the file at a time. Node is a particularly stream-friendly language and ecology, so here a few Node streaming CSV parsers: https://github.com/voodootikigod/node-csv https://github.com/koles/ya-csv https://github.com/lbdremy/node-csv-stream


9

d3.js is also a good javascript library manipulating data. Though most commonly used for building visualizations, you can do any sort of data-driven manipulation in the browser. vega is slightly higher-level visualization grammar built on top of d3.


9

Duplicate reply to a previous question, but you might like this as its all in Python and very easy to automate. I would like to recommend an alternative that I have found that I prefer to open refine. https://github.com/datamade/dedupe It is a very easy to use python program(recommend using in Linux), that provides many custom options to merge your data ...


9

Have you had a look at openstreetmap? http://wiki.openstreetmap.org/wiki/Osmbook Not tested myself, they use Latex to generate the book.


9

The W3C offers a collection of tools that convert from CSV to RDF. However, there is no explicit mention of RDFa in any of these CSV converters. Personally, I'd give the RDF Refine plugin for OpenRefine a try. Once you have your open data in an RDF format, you could use the RDF Translator to turn it into RDFa.


9

Since this question was originally asked, the Dat project has made a lot of progress. Originally conceived by Max Ogden, it now has several other developers working on it. dat is an open source tool that enables the sharing of large datasets, the goal being a collaboration flow similar to what git offers for source code. As a team we have a bias ...


9

I ended up creating my own system combining a bunch of APIs. Here is what I did: I pulled the homepage text from each website and cleaned up all the html, javascript and styles I pushed this text out to various APIs to process and stored the results. I used alchemyapi.com, textrazor.com, opencalais.com. Those APIs have a lot of options but mainly I focused ...


8

Another open-source one that has popped up is DVC https://github.com/iterative/dvc Data Science Version Control or DVC is an open-source tool for data science and machine learning projects. With a simple and flexible Git-like architecture and interface DVC is compatible with Git for storing code and the dependency graph (DAG), but not data files cache. To ...


8

On Windows, there is also a software called Delimit ("Open data files up to 2 billion rows and 2 million columns large!") http://delimitware.com For instance it can split, sort and extract only some rows or columns.


8

You can use DBpedia Spotlight to extract semantic annotations from DBpedia. Here is the code for Python. You will need these libraries: BeautifulSoup urllib2 urllib json The example is only for one link, but you can create a script to itterate through your url list. from bs4 import BeautifulSoup import urllib2 import json import urllib link = "http://...


7

I worked on the creation of a very large (300 million) company authority file for a major publisher and have been involved in proper noun resolution for a number of years. The solution involves two parts: extraction of probable company names and then resolving those names to a standardized "canonical" name. The extraction portion depends upon the source ...


7

learn a general programming language first I would recommend focusing your efforts on learning a general-purpose scripting language suitable for working with data, such as python or R, before fiddling with dedicated tools for very specific purposes. Such languages have a wealth of resources for interacting with APIs, SQL-like databases, generating ...


7

The most versatile tool for geo format conversion is ogr2ogr in gdal. Here's an online front end that uses ogr2ogr to convert to and from GeoJSON, and it supports GeoRSS.


7

Load the file into PostgreSQL database table with a Copy statement. This will give you the full capabilities of SQL syntax, plus the ability to index columns for faster access. For complex queries you have a optimizer that a can figure out the fastest way to access the data. PostgreSQL has smarter I/O than most applications it will detect sequential ...


7

GitHub has all the things, seriously, it rules. OKFN hosts open data there. So do I. As do most Code for America brigades. But you can also get a free datahub.io account, at least for an organization and you can upload there. You should also check out your area. Here in Virginia, there is an OpenVA data portal, and anyone can upload related data after ...


Only top voted, non community-wiki answers of a minimum length are eligible