14

Data.gov has a lot of datasets, and a super-majority of them are mapping. Is there a way to generate a list of all datasets other than geodata?

12

Currently Data.gov is run on bunch of technologies. Specifically one of the primary data platforms is Socrata - explore.data.gov.

There is API for it defined here http://dev.socrata.com/, but essentially you can call this endpoint and iterate through all pages of master list of datasets

https://explore.data.gov/api/dcat.json?page=1

I have sample scrapers for list of datasets and associated metadata here: https://github.com/kachok/data-json/tree/master/data-gov

You then can filter spatial vs tabular data based on metadata (I believe all of the datasets in explore.data.gov are tabular)

With imminent migration of data.gov to CKAN, all of the above will be obsolete (but new API will allow to do similar things as well)

9

The bifurcated view of open data on Data.gov will evaporate later this month and there will be an API to the entire catalog. You can also separate that catalog along geo and non-geo data if that's your preference. To get an early look at the direction we are heading, check out: http://geo.gov.ckan.org/dataset

Raw data is being merged with geo-data as we speak in this demo site. This is a work in progress, so you'll be seeing updates and upgrades as we implement them before we launch operationally with the new catalog in late May 2013.

3

The interactive data sets on Data.gov are currently available here: https://explore.data.gov/catalog/next-gen. I do not they think they are currently available through the basic "raw datasets" https://explore.data.gov/catalog/raw link that Clay gives above.

You may also want to check out this GitHub listing of currently available U.S. government APIs, some of which are not on Data.gov: https://github.com/GSA/slash-developer-pages. It is currently actively maintained.

3

This is an update to the answers above as of January 2015.

I've got the following results when trying the above urls:

Re Dmitry's answer:

https://explore.data.gov/api/dcat.json?page=1

does not render json output, redirects to http://catalog.data.gov/dataset (html listing) instead.

The Socrata Api is not in place any more. So all queries to the Socrata Api endpoint, like:

https://explore.data.gov/catalog/next-gen
https://explore.data.gov/catalog/raw

won't work.

Re Jeanne's answer: the demo site under

http://geo.gov.ckan.org/dataset

is dead and redirects to a more or less empty IP address.

Unfortunately, there seems to be no way to get a single list of dataset ids (or names) via a CKAN package_search query to catalog.data.gov, since the supported solr subset does not allow for the fl-parameter (which would get a subset of fields in each metadata record). Also the ckan legacy rest apis are redirected to package_search. So currently no way to get a list of datasets, whether geo or not.

2

Data.gov divides itself into two categories: raw, and geodata. Geodata, in data.gov terminology, is essentially "shape files" which you can find here:

http://geo.data.gov/geoportal/catalog/main/home.page

The non-geodata is found here:

https://explore.data.gov/catalog/raw It still, however, has data in it that is associated to particular places. Like Toxics Release Inventories by state, or Earthquake plots. That, however, is not "geodata" by Data.gov's terminology.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.