59

This is the kind of thing that the csvkit was built for: csvgrep -c "Healthcare Provider Taxonomy Code_1" -r '^282N' npidata_20050523-20131110.csv > hospitals.csv csvkit is a suite of utilities for converting to and working with CSV, the king of tabular file formats. A little more efficiently, you could do: zcat NPPES_Data_Dissemination_Nov_2013....


54

Here is a guide for each file format from the Open data handbook and a suggestion with a Python library to use. JSON is a simple file format that is very easy for any programming language to read. Its simplicity means that it is generally easier for computers to process than others, such as XML. Working with JSON in Python is almost the same such as working ...


27

On Windows, SweetScape 010 Editor is the best application I am aware of to open/edit large files (easily up to 25 GB). It took around 10 seconds on my computer to open your 4 GB file (SSD): More such tools: Text editor to open big (giant, huge, large) text files


16

As you're only taking a portion of the file, you may be able to use simple tools to subset it before processing. That may get it down to a reasonable size to work with. If you're working on a posix (ie, unix-like) system, you can use shell commands to reduce the file: zcat -cfilename| grep(pattern to match hospitals only)>outputFile This lets you ...


14

you can connect to the file with sql and run your analysis from there. i have written extremely detailed r code (r is free and open source) about how to work with the nppes from your laptop here: http://asdfree.com/national-plan-and-provider-enumeration-system-nppes.html if you have never used r before, check out http://twotorials.com for a crash course ...


12

I surprised no-one mentioned HDF5 yet. This is a documented format with freely available C libraries for using it. It has traditionally been used for high-performance persistence of numerical scientific data. HDF5 files are self-describing and very easy to work with using python (using either PyTables or H5Py). More importantly, HDF5 is platform agnostic and ...


11

Take a look at what OpenStreetMap does. There's a page describing the nature of tile server disk usage. If you go up to zoom level 18 worldwide, you're talking about 91,625,968,981 tiles, which would take around 54000GB of disk space, but would mostly never be viewed. So I'm not sure if it would ever be a sensible approach, but having said that, I heard ...


10

You can find a list of GIS file formats on Wikipedia. Here is a decent overview of open source GIS servers from the gis.se site, these are the servers people who use open data will most likely be using, so target the formats that those servers use. I would consider some kind of open vector/raster format (I like geoJSON for personal projects because it works ...


10

There are streaming CSV parsers, that only look at a small window of the file at a time. Node is a particularly stream-friendly language and ecology, so here a few Node streaming CSV parsers: https://github.com/voodootikigod/node-csv https://github.com/koles/ya-csv https://github.com/lbdremy/node-csv-stream


10

Others have mentioned way to pull apart this file incrementally. It seems to me like you are also commenting on use of resources for a large file. For some solutions you can incrementally read the compressed file uncompressing as you go and feed it through the csv module. For example in python with gzip'd input you would do this by: import csv import ...


10

Ideally, like @Andrew - OpenGeoCode mentions, you would release it in multiple formats. I would really suggest you look into organizing it into a e-book written in Markdown and hosted on Github. There are several advantages to this such as a built-in change log, being able to let people to (publicly) fork your document(s) and share their changes with the ...


9

GTFS is what everyone is using. Other vendors, and even open platforms that consume transit data, consume GTFS. Open Trip Planner is an example. (As an aside: was there a particular need that had that you feel GTFS doesn't address? Perhaps we could give you a more focused answer if you clarified.)


9

ResumeRDF seems to be the most "ontological" approach to normalization of CV information. The W3C has another article about ResumeRDF with further links.


8

Transmodel is a not very widely used format for schedule data (alternative to GTFS). For real time data (alternative to GTFS-realtime): SIRI is an XML protocol used most heavily in Europe. You'll want to consider what formats developers are most aware of and any possible performance issues. TRANSMODEL has been adopted as the European experimental ...


8

Once you've decided on a data format, you will need to decide the following additional issues before publishing your open dataset: Data Layout Name/Data type (Vocabulary) of data fields. Language and Language Script. Data Layout Tabular Data Format (TDF) - http://dataprotocols.org/tabular-data-package/ The key features of this format are the following: •...


8

On Windows, there is also a software called Delimit ("Open data files up to 2 billion rows and 2 million columns large!") http://delimitware.com For instance it can split, sort and extract only some rows or columns.


7

As said in the question, the format used depends mainly on which project/domain the data comes from. But most of the time the data is useless without metadata information, and when the dataset starts to get big you need a way to extract the metadata from it automatically. In astronomy, where most of the data has been open for decades, the International ...


7

I don't think there is such a thing. Near as I can tell, before Google came along and prodded them, a machine-readable transit data format for transit agencies didn't exist. There is a Transit Developers Google Group, you might want to check there as well.


7

Load the file into PostgreSQL database table with a Copy statement. This will give you the full capabilities of SQL syntax, plus the ability to index columns for faster access. For complex queries you have a optimizer that a can figure out the fastest way to access the data. PostgreSQL has smarter I/O than most applications it will detect sequential ...


6

My experience, making maps from quite a few government datasets: For point data, CSV is the best, with "lat" and "lon" columns. Very easy to work with in a wide range of tools, including text editors, spreadsheets, etc. The only downside I've come across so far is that GDAL sometimes requires you to make a .vrt companion file. (EDIT: Another downside is ...


6

Every standard is intended to serve a slightly different purpose. Even if we're talking messaging standards over HTTP, you have both REST and SOAP. (and before I get all of the SOAP haters commenting ... there is a ton of bad SOAP implementations, but not everything is documented oriented and meshes well with REST) Before we had SOAP there were standards ...


6

The most versatile tool for geo format conversion is ogr2ogr in gdal. Here's an online front end that uses ogr2ogr to convert to and from GeoJSON, and it supports GeoRSS.


6

This answer is not really useful for non-programmers, but if could manage some programming in perl, the Parse::CSV module is especially designed for this task. From the doc: It provides a flexible and light-weight streaming parser for large, extremely large, or arbitrarily large CSV files. Perl is usually very good for data mining tasks.


6

The other suggestions on here are great, and I would echo the recommendation to consider Github if you want people to be able to easily edit your product for creating their own versions. This will also help you track the edits that have been made so you can learn more about what people do with your manuals. If you want to learn more about the definitions ...


6

The format should fit the data structure not the need of the comsumer (or the one who uses the data). If the data is semantically stored in a json structure you can easily export the data in different formats. So my answer is: json Check https://schema.org/Event


5

Apart from MapBox mentioned by Harry I'd also recommend having a look at CartoDB which let's you harness the power of PosGIS without the hassle of maintaining your own server (at a price). Alternative solution is to get your own (virtual) server running and equip it with a 'geostack' of map server with database and then start building applications using map ...


5

Have a look at Open MapQuest. They offer the Nasa BlueMarble, Satellite and different versions of OSM for free. They only require you to attribute them and OSM, etc., and let them know if your app will exceed 4000 tiles per second. You could set up your own Geoserver instance, Windows based or whatever (it's a self contained Java app), use mapserver as a ...


5

I have used utilities such as (g)awk to readlarge file such as this record by record. I the extract the required information from each line and write it to an output file. For windows users (g)awk is available in cygwin. I have also used python to achieve the same result. You could implement this process in most programming languages.


5

Well, in short, Talend Open Studio for Data Integration is an ETL. It can be used for many use cases, including data migration, files processing, etc. You can easily build jobs using a visual editor to combine specialized connectors (read CSV files, select rows corresponding to your criteria, write result to one or more files or directly to a database, and ...


5

If you're on Windows, I can't sing the praises of LogParser high enough. It allows you to query files in a wide variety of formats (mostly log formats as that's what it was meant for, but XML and CSV are valid). You query the file with a surprisingly complete SQL syntax, and you can even use it to import an entire file directly into an SQL database very ...


Only top voted, non community-wiki answers of a minimum length are eligible