This is the kind of thing that the csvkit was built for:
csvgrep -c "Healthcare Provider Taxonomy Code_1" -r '^282N' npidata_20050523-20131110.csv > hospitals.csv
csvkit is a suite of utilities for converting to and working with CSV, the king of tabular file formats.
A little more efficiently, you could do:
Basic answer afaik is: "No".
As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures.
Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as ...
Supported input data formats for tabular data are (as of August 2015): CSV, TSV, JSON, and newline delimited JSON. It also supports ...
On Windows, SweetScape 010 Editor is the best application I am aware of to open/edit large files (easily up to 25 GB). It took around 10 seconds on my computer to open your 4 GB file (SSD):
More such tools: Text editor to open big (giant, huge, large) text files
Please note that git has two configuration commands:
git config filter.<driver>.clean
git config filter.<driver>.smudge
The clean filter can be used e.g. to sort all records in a CSV file except the top row, thus making re-ordering irrelevant.
Quoting gitattributes(5) :
A filter driver consists of a clean command and a smudge command, either of ...
As you're only taking a portion of the file, you may be able to use simple tools to subset it before processing. That may get it down to a reasonable size to work with.
If you're working on a posix (ie, unix-like) system, you can use shell commands to reduce the file:
zcat -cfilename| grep(pattern to match hospitals only)>outputFile
This lets you ...
Can you provide a more comprehensive list of the pain points you're experiencing with Git? Git works great for data.
In addition to the config flag Deer Hunter mentioned for sorting, you can also teach git to diff on a word-by-word (rather than line-by-line basis), by simply passing the --word-diff flag.
you can connect to the file with sql and run your analysis from there.
i have written extremely detailed r code (r is free and open source) about how to work with the nppes from your laptop here:
if you have never used r before, check out http://twotorials.com for a crash course
Here is my take on it: I use R and its IDE RStudio.
The hard part, cleaning the data, is luckily done. Sharing the CSV via a dropbox link is not bad. The file is well structured. To improve it you could add a licence and provide a bit more information about the source. For more information see our certificates.
If you want to publish in a more "...
You can use Google Spreadsheets ImportHTML formula as detailed in this Liberating HTML Tables (using Google Spreadsheet) tutorial on School of Data by Tony Hirst - it includes a specific walk-through for Wikipedia.
The essence is to do:
In your case you could try:
There is no simple answer to this question, because ZIP codes do not represent geographical areas. They represent postal delivery routes, which are sometimes simply a bank of PO boxes in a specific post office, and are sometimes an organization like a University which has its own internal mail processing services.
Therefore, not all ZIP Codes can truly be ...
Others have mentioned way to pull apart this file incrementally. It seems to me like you are also commenting on use of resources for a large file. For some solutions you can incrementally read the compressed file uncompressing as you go and feed it through the csv module. For example in python with gzip'd input you would do this by:
There has been some mention about this being a complex problem. It's not hard at all. In fact, matching a zipcode to the corresponding cities is super simple. Granted, trying to find an accurate shapefile that corresponds to a given zipcode, that is a more challenging issue. But simply matching zipcode to city/cities... very easy. The USPS data has a ...
There are streaming CSV parsers, that only look at a small window of the file at a time.
Node is a particularly stream-friendly language and ecology, so here a few Node streaming CSV parsers:
vega is slightly higher-level visualization grammar built on top of d3.
Duplicate reply to a previous question, but you might like this as its all in Python and very easy to automate.
I would like to recommend an alternative that I have found that I prefer to open refine.
It is a very easy to use python program(recommend using in Linux), that provides many custom options to merge your data ...
The W3C offers a collection of tools that convert from CSV to RDF. However, there is no explicit mention of RDFa in any of these CSV converters. Personally, I'd give the RDF Refine plugin for OpenRefine a try.
Once you have your open data in an RDF format, you could use the RDF Translator to turn it into RDFa.
Since this question was originally asked, the Dat project has made a lot of progress. Originally conceived by Max Ogden, it now has several other developers working on it.
dat is an open source tool that enables the sharing of large
datasets, the goal being a collaboration flow similar to what git
offers for source code. As a team we have a bias ...
I ended up creating my own system combining a bunch of APIs. Here is what I did:
I pushed this text out to various APIs to process and stored the results. I used alchemyapi.com, textrazor.com, opencalais.com. Those APIs have a lot of options but mainly I focused ...
Another open-source one that has popped up is DVC
Data Science Version Control or DVC is an open-source tool for data science and machine learning projects. With a simple and flexible Git-like architecture and interface
DVC is compatible with Git for storing code and the dependency graph (DAG), but not data files cache. To ...
On Windows, there is also a software called Delimit ("Open data files up to 2 billion rows and 2 million columns large!") http://delimitware.com For instance it can split, sort and extract only some rows or columns.
You can use DBpedia Spotlight to extract semantic annotations from DBpedia. Here is the code for Python.
You will need these libraries:
The example is only for one link, but you can create a script to itterate through your url list.
from bs4 import BeautifulSoup
link = "http://...
I worked on the creation of a very large (300 million) company authority file for a major publisher and have been involved in proper noun resolution for a number of years. The solution involves two parts: extraction of probable company names and then resolving those names to a standardized "canonical" name. The extraction portion depends upon the source ...
learn a general programming language first
I would recommend focusing your efforts on learning a general-purpose scripting language suitable for working with data, such as python or R, before fiddling with dedicated tools for very specific purposes. Such languages have a wealth of resources for interacting with APIs, SQL-like databases, generating ...
Load the file into PostgreSQL database table with a Copy statement. This will give you the full capabilities of SQL syntax, plus the ability to index columns for faster access.
For complex queries you have a optimizer that a can figure out the fastest way to access the data. PostgreSQL has smarter I/O than most applications it will detect sequential ...
GitHub has all the things, seriously, it rules. OKFN hosts open data there. So do I. As do most Code for America brigades. But you can also get a free datahub.io account, at least for an organization and you can upload there.
You should also check out your area. Here in Virginia, there is an OpenVA data portal, and anyone can upload related data after ...