How do I get a full list of datasets available on Data.Gov using the CKAN API?

Question

I'm fairly new to python, but I've been struggling to get anything more than ten results from a package_list request from Data.Gov's CKAN API. Here's what I'm using:

response = urlopen('http://catalog.data.gov/api/3/action/package_list')
assert response.code == 200

response_dict = json.loads(response.read())

assert response_dict['success'] is True
len(response_dict['result']['results'])

The result is only 10 datasets. This is the same when I use package_search, it only returns the first ten results. What am I doing wrong? I can't seem to find the answer in the CKAN documentation.

Any suggestions/help would be appreciated!

• Ben

Answer 1

For now the Data.gov CKAN API redirects package_list to package_search and for package_search the relevant Solr parameters to limit your query are rows and start.

For example, http://catalog.data.gov/api/3/action/package_search?rows=1000&start=0

And then page through.

If you or anyone has additional feedback on what Data.gov CKAN API documentation would be helpful or would like to help contribute there is a relevant issue open on GitHub.

Answer 2

The API documentation lists two arguments to package_list :

Parameters:

• limit (int) – if given, the list of datasets will be broken into pages of at most limit datasets per page and only one page will be returned at a time (optional)
• offset (int) – when limit is given, the offset to start returning packages from

They don't mention if they count from 1 or from 0 ... but I'd suggest trying increasing limit sizes (100, 1000, 5000, etc.) until they stop returning that many, and then send an offset of the number they gave you, and see if they still give you more, then keep adding limit to the offset value until they don't give you anything.