Data.gov has a lot of datasets, and a super-majority of them are mapping. Is there a way to generate a list of all datasets other than geodata?
Add a comment
|
5 Answers
Currently Data.gov is run on bunch of technologies. Specifically one of the primary data platforms is Socrata - explore.data.gov.
There is API for it defined here http://dev.socrata.com/, but essentially you can call this endpoint and iterate through all pages of master list of datasets
https://explore.data.gov/api/dcat.json?page=1
I have sample scrapers for list of datasets and associated metadata here: https://github.com/kachok/data-json/tree/master/data-gov
You then can filter spatial vs tabular data based on metadata (I believe all of the datasets in explore.data.gov are tabular)
With imminent migration of data.gov to CKAN, all of the above will be obsolete (but new API will allow to do similar things as well)
The bifurcated view of open data on Data.gov will evaporate later this month and there will be an API to the entire catalog. You can also separate that catalog along geo and non-geo data if that's your preference. To get an early look at the direction we are heading, check out: http://geo.gov.ckan.org/dataset
Raw data is being merged with geo-data as we speak in this demo site. This is a work in progress, so you'll be seeing updates and upgrades as we implement them before we launch operationally with the new catalog in late May 2013.
answered May 10, 2013 at 15:30
The interactive data sets on Data.gov are currently available here: https://explore.data.gov/catalog/next-gen. I do not they think they are currently available through the basic "raw datasets" https://explore.data.gov/catalog/raw link that Clay gives above.
You may also want to check out this GitHub listing of currently available U.S. government APIs, some of which are not on Data.gov: https://github.com/GSA/slash-developer-pages. It is currently actively maintained.
This is an update to the answers above as of January 2015.
I've got the following results when trying the above urls:
Re Dmitry's answer:
https://explore.data.gov/api/dcat.json?page=1
does not render json output, redirects to http://catalog.data.gov/dataset (html listing) instead.
The Socrata Api is not in place any more. So all queries to the Socrata Api endpoint, like:
https://explore.data.gov/catalog/next-gen
https://explore.data.gov/catalog/raw
won't work.
Re Jeanne's answer: the demo site under
http://geo.gov.ckan.org/dataset
is dead and redirects to a more or less empty IP address.
Unfortunately, there seems to be no way to get a single list of dataset ids (or names) via a CKAN package_search query to catalog.data.gov, since the supported solr subset does not allow for the fl-parameter (which would get a subset of fields in each metadata record). Also the ckan legacy rest apis are redirected to package_search. So currently no way to get a list of datasets, whether geo or not.
Data.gov divides itself into two categories: raw, and geodata. Geodata, in data.gov terminology, is essentially "shape files" which you can find here:
http://geo.data.gov/geoportal/catalog/main/home.page
The non-geodata is found here:
https://explore.data.gov/catalog/raw It still, however, has data in it that is associated to particular places. Like Toxics Release Inventories by state, or Earthquake plots. That, however, is not "geodata" by Data.gov's terminology.
