We’re rewarding the question askers & reputations are being recalculated! Read more.
59

This is the kind of thing that the csvkit was built for: csvgrep -c "Healthcare Provider Taxonomy Code_1" -r '^282N' npidata_20050523-20131110.csv > hospitals.csv csvkit is a suite of utilities for converting to and working with CSV, the king of tabular file formats. A little more efficiently, you could do: zcat NPPES_Data_Dissemination_Nov_2013....


41

Basic answer afaik is: "No". As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures. Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as ...


35

I recently stumbled on this article by the Open Knowledge Foundation regarding the design of a graphical interface to diff tabular data called daff. It can also be tested and forked on GitHub.


29

There is a project dat that addresses what you are looking for. It is now in beta, featuring a command line tool with similar design like git. There are also APIs for Javascript (dat-core), Python (datpy) and R (rdat). Supported input data formats for tabular data are (as of August 2015): CSV, TSV, JSON, and newline delimited JSON. It also supports ...


27

On Windows, SweetScape 010 Editor is the best application I am aware of to open/edit large files (easily up to 25 GB). It took around 10 seconds on my computer to open your 4 GB file (SSD): More such tools: Text editor to open big (giant, huge, large) text files


25

There are actually quite a few applications for visualizing and analyzing graphs: Gephi and Cytoscape are two well-known open source applications that support large and complex graphs. If you're mainly interested in visualizing graphs, have a look at Graphviz, which is an absolute classic. You can also use R or commercial tools like Mathematica if you're ...


22

OpenRefine can be used to parse semi-structured data into a table like structure, where it can be operated on in a manner similar to a spread sheet and exported. The site features a tutorial on converting a list on wikipedia into a table which may be a good starting point. The operations involved can also be exported incase you need to perform the same ...


20

[Note: I'm the co-founder of OpenCorporates, which along with our reconciliation service for OpenRefine, has been kindly mentioned by several of the answers, but I've tried to cover some of the general issues here, using our experience, rather than suggesting we've got all the answers] This is a really difficult problem because in general it requires more ...


19

Please note that git has two configuration commands: git config filter.<driver>.clean git config filter.<driver>.smudge The clean filter can be used e.g. to sort all records in a CSV file except the top row, thus making re-ordering irrelevant. Quoting gitattributes(5) : A filter driver consists of a clean command and a smudge command, ...


18

CKAN stands for Comprehensive Knowledge Archive Network. CKAN is a self-described data portal platform that allows an organization to manage, publish, and share data and for others to find and use that data. In general, data portal platforms provide a structured solution of software, policies, and guidelines that let an organization (often a government ...


16

As you're only taking a portion of the file, you may be able to use simple tools to subset it before processing. That may get it down to a reasonable size to work with. If you're working on a posix (ie, unix-like) system, you can use shell commands to reduce the file: zcat -cfilename| grep(pattern to match hospitals only)>outputFile This lets you ...


15

Project DBpedia is a crowd-sourced community effort to extract structured information from Wikipedia. Drugs and Chemicals infoboxes are available in structured form, already. Check out http://dbpedia.org/page/Rosiglitazone


15

Python is the tool I would use. import csv from itertools import izip_longest def grouper(iterable, n, fillvalue=None): '''Collect data into fixed-length chunks or blocks grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx''' args = [iter(iterable)] * n return izip_longest(fillvalue=fillvalue, *args) with open("raw_data.txt", 'r') as f: ...


15

OpenRefine (formerly Google Refine) offers nice tools for data cleansing, e.g. correcting slight spelling variations. You can also script all transformations on the data and re-apply them later for updated datasets. Is this what you're looking for?


15

Can you provide a more comprehensive list of the pain points you're experiencing with Git? Git works great for data. In addition to the config flag Deer Hunter mentioned for sorting, you can also teach git to diff on a word-by-word (rather than line-by-line basis), by simply passing the --word-diff flag.


14

I found OpenRefine really useful for people new to Open Data. It offer a nice and clean spreadsheet like interface with the power of querying by field (like SQL).


14

I have had great luck with https://github.com/jazzido/tabula Once the PDF is loaded into the system, it takes manual selection of the table to get the data, but I really prefer it over rolling my own computer vision system, as I've found tabula to be highly accurate, and I can't say the same of a 100% automated system.


14

you can connect to the file with sql and run your analysis from there. i have written extremely detailed r code (r is free and open source) about how to work with the nppes from your laptop here: http://asdfree.com/national-plan-and-provider-enumeration-system-nppes.html if you have never used r before, check out http://twotorials.com for a crash course ...


14

Data Here is my take on it: I use R and its IDE RStudio. The hard part, cleaning the data, is luckily done. Sharing the CSV via a dropbox link is not bad. The file is well structured. To improve it you could add a licence and provide a bit more information about the source. For more information see our certificates. If you want to publish in a more "...


13

If you already have a nicely formatted CSV then why not publish that? You can publish in both formats if you really want to do the RDFa too.


12

OpenRefine has come a long way in the recent years. It has become quite flexible, there are many plugins available, and even though there's no programmatic "batch mode", the community is already talking about adding one (see the last entry at the OpenRefine FAQ). It might be beneficial to look into OpenRefine (in combination with OpenCorporates) once more – ...


11

One that just appeared is https://plot.ly. There are many more. Which program is most useful to you depends on a lot of factors. If you are technical proficient, you might like Weka (http://www.cs.waikato.ac.nz/ml/weka/).


11

There is no simple answer to this question, because ZIP codes do not represent geographical areas. They represent postal delivery routes, which are sometimes simply a bank of PO boxes in a specific post office, and are sometimes an organization like a University which has its own internal mail processing services. Therefore, not all ZIP Codes can truly be ...


10

There are also a number of network analysis packages for the open source R language, such as network and sna, and igraph, all of which have some viz capabilities. R can also be a good environment for general data manipulation tasks.


10

Wikipedia offers two interesting ways to get its own information: Complete database dumps in XML and SQL, as you wish. Special export very nice XML files downloadabe from only the categories that you specify. Images and uploaded files are stored elsewhere, also downloadable This is a XML file from the page requested using special export, the Wikipedia ...


10

There's discussion of exactly this in this question on School of Data Q&A site. Among other items mentioned there are (all free/open source): http://coolwanglu.github.io/pdf2htmlEX/ - open-source, looks good but I've not tested for tabular data http://tabula.nerdpower.org/ - open-source, designed specifically for tabular data but looks a bit of a pain ...


10

Figuring out the cost for running CKAN boils down to two different categories: the cost of running the software itself, as well as the cost of building and maintaining your open data catalog going forward. The cost of the software itself is simpler to answer: CKAN itself is free/open-source, and available as a hosted solution. If you go the open-source ...


10

You can use Google Spreadsheets ImportHTML formula as detailed in this Liberating HTML Tables (using Google Spreadsheet) tutorial on School of Data by Tony Hirst - it includes a specific walk-through for Wikipedia. The essence is to do: =importHTML("","table",N) In your case you could try: =importHTML("http://en.wikipedia.org/wiki/...


10

There has been some mention about this being a complex problem. It's not hard at all. In fact, matching a zipcode to the corresponding cities is super simple. Granted, trying to find an accurate shapefile that corresponds to a given zipcode, that is a more challenging issue. But simply matching zipcode to city/cities... very easy. The USPS data has a ...


10

There are streaming CSV parsers, that only look at a small window of the file at a time. Node is a particularly stream-friendly language and ecology, so here a few Node streaming CSV parsers: https://github.com/voodootikigod/node-csv https://github.com/koles/ya-csv https://github.com/lbdremy/node-csv-stream


Only top voted, non community-wiki answers of a minimum length are eligible