I'm trying to harvest data from the CKAN API of Data.gov, using Python code. Doing this however, I ran into problems. I use the following URL:
And increment the 'start' parameter after every request. I have run into two difficulties:
- It seems like I get throttled: Sometimes after a few pages or even at the first page, and other times only after 120 pages (1000 results each) I experience request timeouts, and progress becomes extremely slow (e.g. 100 kb/s). I do have an apiKey for api.data.gov (Is this actually valid for the CKAN API?), but if I include this as 'Authorization' in the http header, still timeouts occur.
- Using an offset of 42000, often only 998 packages are returned, although it is still far from the end of the list (192.633). This occurs by doing the following request: http://catalog.data.gov/api/3/action/package_search?rows=1000&start=42000. Other times it actually does return 1000 results here, but returns 998 results on a later page...
Can I do anything on my end to prevent the throttling from occuring, or is something else wrong?