62

This is the kind of thing that the csvkit was built for: csvgrep -c "Healthcare Provider Taxonomy Code_1" -r '^282N' npidata_20050523-20131110.csv > hospitals.csv csvkit is a suite of utilities for converting to and working with CSV, the king of tabular file formats. A little more efficiently, you could do: zcat NPPES_Data_Dissemination_Nov_2013....


41

I recently stumbled on this article by the Open Knowledge Foundation regarding the design of a graphical interface to diff tabular data called daff. It can also be tested and forked on GitHub.


26

On Windows, SweetScape 010 Editor is the best application I am aware of to open/edit large files (easily up to 25 GB). It took around 10 seconds on my computer to open your 4 GB file (SSD): More such tools: Text editor to open big (giant, huge, large) text files


17

As you're only taking a portion of the file, you may be able to use simple tools to subset it before processing. That may get it down to a reasonable size to work with. If you're working on a posix (ie, unix-like) system, you can use shell commands to reduce the file: zcat -cfilename| grep(pattern to match hospitals only)>outputFile This lets you ...


14

you can connect to the file with sql and run your analysis from there. i have written extremely detailed r code (r is free and open source) about how to work with the nppes from your laptop here: http://asdfree.com/national-plan-and-provider-enumeration-system-nppes.html if you have never used r before, check out http://twotorials.com for a crash course ...


14

Data Here is my take on it: I use R and its IDE RStudio. The hard part, cleaning the data, is luckily done. Sharing the CSV via a dropbox link is not bad. The file is well structured. To improve it you could add a licence and provide a bit more information about the source. For more information see our certificates. If you want to publish in a more "...


11

Others have mentioned way to pull apart this file incrementally. It seems to me like you are also commenting on use of resources for a large file. For some solutions you can incrementally read the compressed file uncompressing as you go and feed it through the csv module. For example in python with gzip'd input you would do this by: import csv import ...


10

There are streaming CSV parsers, that only look at a small window of the file at a time. Node is a particularly stream-friendly language and ecology, so here a few Node streaming CSV parsers: https://github.com/voodootikigod/node-csv https://github.com/koles/ya-csv https://github.com/lbdremy/node-csv-stream


9

Since this question was originally asked, the Dat project has made a lot of progress. Originally conceived by Max Ogden, it now has several other developers working on it. dat is an open source tool that enables the sharing of large datasets, the goal being a collaboration flow similar to what git offers for source code. As a team we have a bias ...


9

I ended up creating my own system combining a bunch of APIs. Here is what I did: I pulled the homepage text from each website and cleaned up all the html, javascript and styles I pushed this text out to various APIs to process and stored the results. I used alchemyapi.com, textrazor.com, opencalais.com. Those APIs have a lot of options but mainly I focused ...


8

Another open-source one that has popped up is DVC https://github.com/iterative/dvc Data Science Version Control or DVC is an open-source tool for data science and machine learning projects. With a simple and flexible Git-like architecture and interface DVC is compatible with Git for storing code and the dependency graph (DAG), but not data files cache. To ...


8

On Windows, there is also a software called Delimit ("Open data files up to 2 billion rows and 2 million columns large!") http://delimitware.com For instance it can split, sort and extract only some rows or columns.


8

You can use DBpedia Spotlight to extract semantic annotations from DBpedia. Here is the code for Python. You will need these libraries: BeautifulSoup urllib2 urllib json The example is only for one link, but you can create a script to itterate through your url list. from bs4 import BeautifulSoup import urllib2 import json import urllib link = "http://...


7

The most versatile tool for geo format conversion is ogr2ogr in gdal. Here's an online front end that uses ogr2ogr to convert to and from GeoJSON, and it supports GeoRSS.


7

Load the file into PostgreSQL database table with a Copy statement. This will give you the full capabilities of SQL syntax, plus the ability to index columns for faster access. For complex queries you have a optimizer that a can figure out the fastest way to access the data. PostgreSQL has smarter I/O than most applications it will detect sequential ...


7

GitHub has all the things, seriously, it rules. OKFN hosts open data there. So do I. As do most Code for America brigades. But you can also get a free datahub.io account, at least for an organization and you can upload there. You should also check out your area. Here in Virginia, there is an OpenVA data portal, and anyone can upload related data after ...


7

Maybe the rcrossref package for R is helpful. To find the number of citations, You can do things such as library(rcrossref) # returns list: # pap <- cr_works(dois = "10.1371/journal.pone.0042793") # pap$data will be 28-column data frame. cr_citation_count(doi = "10.1371/journal.pone.0042793") Result: doi ...


7

Socrata actually isn't built on top of an existing spreadsheet application or platform. We've built our own platform using a number of different open source technologies. Customer data is stored in our backend, where we use different datastores in a sharded configuration to make sure queries are performant while data remains highly available and safe. Long ...


6

If your main focus is to visualize the data and have graph metrics I would recommend this list. If you want an off the shelf package Gephi - Desktop application, Open Source license Cytoscape - Desktop Application - Open Source license Pajek - Desktop Application, Free for non Commercial use (old but still good - I've seen papers using it just few days ago)...


6

The wikitextparser Python module has good support for converting tables into structured formats. Example: import wikitextparser as wtp import requests import json r = requests.get('http://en.wikipedia.org/w/index.php?title=' + 'List_of_country_calling_codes&action=raw') wt = wtp.parse(r.text) print(json.dumps({'root': wt.tables[1].getdata()[0:3]}, ...


6

This answer is not really useful for non-programmers, but if could manage some programming in perl, the Parse::CSV module is especially designed for this task. From the doc: It provides a flexible and light-weight streaming parser for large, extremely large, or arbitrarily large CSV files. Perl is usually very good for data mining tasks.


6

If you just want a simple GUI to interact with shapefiles (plot maps, see properties and do some manipulation) you can use the free/libre software QGIS. If you want more control and a programmable interface you can go with Python using libraries such as fiona and shapely. And if you want to display your shapes in the Web, I highly recommend the JavaScript ...


6

I've compiled a fairly extensive list of open government portals both in US and around the world in the last few days. I am working on code to support crowdsourcing the catalog. I also have a CSV version for download. There is about 700 sites listed so far: ONLINE CATALOG: http://www.opengeocode.org/opendata/ DOWNLOAD: http://www.opengeocode.org/opendata/...


6

tamu has a killer geocoder, as well as a definitive list of other free geocoders. all you have to do is upload your data http://geoservices.tamu.edu/Services/Geocode/BatchProcess/ http://geoservices.tamu.edu/Services/Geocode/OtherGeocoders/


5

Yes there is one, it is called noms https://github.com/attic-labs/noms Noms is a decentralized database philosophically descendant from the Git version control system.


5

My team builds https://vida.io a tool for creating data visualization templates. We support d3.js templates. You can see a lot of examples on our site: https://vida.io/explore For simple chart/visualization, you can use Google visualization tool: https://vida.io/pages/google-charts


5

There are a few solutions to visualize graphs : D3.js, Sigma.js, KeyLines, Gephi, Linkurious, Neoclipse, Neovigator. Here is a table that compare some of these options : http://linkurio.us/comparative-study/ And a quick presentation about the different approaches to visualizing graphs : http://www.slideshare.net/Linkurious/graph-visualization-options-and-...


5

As far as I can see, there is no link from the category Japanese people to the article John Andru (at least based on dump from 1 October 2013), so I'm not sure why did the tool tell you otherwise. But the category structure is pretty messed up. For example, John Andru is in the category 18th-century German writers, through chain of categories like this: ...


5

I have used utilities such as (g)awk to readlarge file such as this record by record. I the extract the required information from each line and write it to an output file. For windows users (g)awk is available in cygwin. I have also used python to achieve the same result. You could implement this process in most programming languages.


5

Well, in short, Talend Open Studio for Data Integration is an ETL. It can be used for many use cases, including data migration, files processing, etc. You can easily build jobs using a visual editor to combine specialized connectors (read CSV files, select rows corresponding to your criteria, write result to one or more files or directly to a database, and ...


Only top voted, non community-wiki answers of a minimum length are eligible