This is the kind of thing that the csvkit was built for:
csvgrep -c "Healthcare Provider Taxonomy Code_1" -r '^282N' npidata_20050523-20131110.csv > hospitals.csv
csvkit is a suite of utilities for converting to and working with CSV, the king of tabular file formats.
A little more efficiently, you could do:
Basic answer afaik is: "No".
As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures.
Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as ...
Supported input data formats for tabular data are (as of August 2015): CSV, TSV, JSON, and newline delimited JSON. It also supports ...
On Windows, SweetScape 010 Editor is the best application I am aware of to open/edit large files (easily up to 25 GB). It took around 10 seconds on my computer to open your 4 GB file (SSD):
More such tools: Text editor to open big (giant, huge, large) text files
There are actually quite a few applications for visualizing and analyzing graphs:
Gephi and Cytoscape are two well-known open source
applications that support large and complex graphs.
If you're mainly interested in visualizing graphs, have a look at
Graphviz, which is an absolute classic.
You can also use R or commercial tools like Mathematica if you're
OpenRefine can be used to parse semi-structured data into a table like structure, where it can be operated on in a manner similar to a spread sheet and exported.
The site features a tutorial on converting a list on wikipedia into a table which may be a good starting point.
The operations involved can also be exported incase you need to perform the same ...
[Note: I'm the co-founder of OpenCorporates, which along with our reconciliation service for OpenRefine, has been kindly mentioned by several of the answers, but I've tried to cover some of the general issues here, using our experience, rather than suggesting we've got all the answers]
This is a really difficult problem because in general it requires more ...
Please note that git has two configuration commands:
git config filter.<driver>.clean
git config filter.<driver>.smudge
The clean filter can be used e.g. to sort all records in a CSV file except the top row, thus making re-ordering irrelevant.
Quoting gitattributes(5) :
A filter driver consists of a clean command and a smudge command, either of ...
CKAN stands for Comprehensive Knowledge Archive Network. CKAN is a self-described data portal platform that allows an organization to manage, publish, and share data and for others to find and use that data.
In general, data portal platforms provide a structured solution of software, policies, and guidelines that let an organization (often a government ...
As you're only taking a portion of the file, you may be able to use simple tools to subset it before processing. That may get it down to a reasonable size to work with.
If you're working on a posix (ie, unix-like) system, you can use shell commands to reduce the file:
zcat -cfilename| grep(pattern to match hospitals only)>outputFile
This lets you ...
Project DBpedia is a crowd-sourced community effort to extract structured information from Wikipedia.
Drugs and Chemicals infoboxes are available in structured form, already.
Python is the tool I would use.
from itertools import izip_longest
def grouper(iterable, n, fillvalue=None):
'''Collect data into fixed-length chunks or blocks
grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx'''
args = [iter(iterable)] * n
return izip_longest(fillvalue=fillvalue, *args)
with open("raw_data.txt", 'r') as f:
I have had great luck with https://github.com/jazzido/tabula
Once the PDF is loaded into the system, it takes manual selection of the table to get the data, but I really prefer it over rolling my own computer vision system, as I've found tabula to be highly accurate, and I can't say the same of a 100% automated system.
OpenRefine (formerly Google Refine) offers nice tools for data cleansing, e.g. correcting slight spelling variations. You can also script all transformations on the data and re-apply them later for updated datasets.
Is this what you're looking for?
Can you provide a more comprehensive list of the pain points you're experiencing with Git? Git works great for data.
In addition to the config flag Deer Hunter mentioned for sorting, you can also teach git to diff on a word-by-word (rather than line-by-line basis), by simply passing the --word-diff flag.
you can connect to the file with sql and run your analysis from there.
i have written extremely detailed r code (r is free and open source) about how to work with the nppes from your laptop here:
if you have never used r before, check out http://twotorials.com for a crash course
Here is my take on it: I use R and its IDE RStudio.
The hard part, cleaning the data, is luckily done. Sharing the CSV via a dropbox link is not bad. The file is well structured. To improve it you could add a licence and provide a bit more information about the source. For more information see our certificates.
If you want to publish in a more "...
OpenRefine has come a long way in the recent years. It has become quite flexible, there are many plugins available, and even though there's no programmatic "batch mode", the community is already talking about adding one (see the last entry at the OpenRefine FAQ).
It might be beneficial to look into OpenRefine (in combination with OpenCorporates) once more – ...
You can use Google Spreadsheets ImportHTML formula as detailed in this Liberating HTML Tables (using Google Spreadsheet) tutorial on School of Data by Tony Hirst - it includes a specific walk-through for Wikipedia.
The essence is to do:
In your case you could try:
One that just appeared is https://plot.ly. There are many more. Which program is most useful to you depends on a lot of factors. If you are technical proficient, you might like Weka (http://www.cs.waikato.ac.nz/ml/weka/).
There is no simple answer to this question, because ZIP codes do not represent geographical areas. They represent postal delivery routes, which are sometimes simply a bank of PO boxes in a specific post office, and are sometimes an organization like a University which has its own internal mail processing services.
Therefore, not all ZIP Codes can truly be ...
Others have mentioned way to pull apart this file incrementally. It seems to me like you are also commenting on use of resources for a large file. For some solutions you can incrementally read the compressed file uncompressing as you go and feed it through the csv module. For example in python with gzip'd input you would do this by:
There are also a number of network analysis packages for the open source R language, such as network and sna, and igraph, all of which have some viz capabilities. R can also be a good environment for general data manipulation tasks.
Wikipedia offers two interesting ways to get its own information:
Complete database dumps in XML and SQL, as you wish.
Special export very nice XML files downloadabe from only the categories that you specify.
Images and uploaded files are stored elsewhere, also downloadable
This is a XML file from the page requested using special export, the Wikipedia ...
There's discussion of exactly this in this question on School of Data Q&A site.
Among other items mentioned there are (all free/open source):
http://coolwanglu.github.io/pdf2htmlEX/ - open-source, looks good but I've not tested for tabular data
http://tabula.nerdpower.org/ - open-source, designed specifically for tabular data but looks a bit of a pain ...
Figuring out the cost for running CKAN boils down to two different categories: the cost of running the software itself, as well as the cost of building and maintaining your open data catalog going forward.
The cost of the software itself is simpler to answer: CKAN itself is free/open-source, and available as a hosted solution.
If you go the open-source ...
There has been some mention about this being a complex problem. It's not hard at all. In fact, matching a zipcode to the corresponding cities is super simple. Granted, trying to find an accurate shapefile that corresponds to a given zipcode, that is a more challenging issue. But simply matching zipcode to city/cities... very easy. The USPS data has a ...