7

I'm trying to harvest data from the CKAN API of Data.gov, using Python code. Doing this however, I ran into problems. I use the following URL:

http://catalog.data.gov/api/3/action/package_search?rows=1000&start=0

And increment the 'start' parameter after every request. I have run into two difficulties:

  1. It seems like I get throttled: Sometimes after a few pages or even at the first page, and other times only after 120 pages (1000 results each) I experience request timeouts, and progress becomes extremely slow (e.g. 100 kb/s). I do have an apiKey for api.data.gov (Is this actually valid for the CKAN API?), but if I include this as 'Authorization' in the http header, still timeouts occur.
  2. Using an offset of 42000, often only 998 packages are returned, although it is still far from the end of the list (192.633). This occurs by doing the following request: http://catalog.data.gov/api/3/action/package_search?rows=1000&start=42000. Other times it actually does return 1000 results here, but returns 998 results on a later page...

Can I do anything on my end to prevent the throttling from occuring, or is something else wrong?

Improve this question
4
  • 1
    Did you consider/try already the CSW endpoint (see docs at cms.geoplatform.gov/csw-resources). To harvest CSW with Python there's OWSLib. Doesn't appear to be a need for any API key when I tried
    – nmtoken
    Mar 18 '17 at 9:44
  • 1
    Have just taken a look at the csw endpoint, but doesn't look too stable either. The basic request https://catalog.data.gov/csw?service=CSW&version=2.0.2&request=GetCapabilities only works sometimes, now and then returns an xml exception-report for me: Could not load repository (local): (OperationalError) SSL SYSCALL error: EOF detected "\n SELECT c.oid\n FROM pg_catalog.pg_class c\n LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace\n WHERE (pg_catalog.pg_table_is_visible(c.oid))\n AND c.relname = %(table_name)s AND c.relkind in ('r','v')\n " {'table_name': u'records'}
    – Tom Brouwer
    Mar 19 '17 at 14:09
  • Additionally: I used the GetRecords function, as described on (this page)[gist.github.com/kalxas/5ab6237b4163b0fdc930]. This returns only 10 records, which already takes multiple seconds for me, and does not contain the full suite of metadata elements, the CKAN API returns.
    – Tom Brouwer
    Mar 19 '17 at 14:16
  • I guess I was lucky when I tried the CSW interface (didn't get the error), it might be worth reporting as a bug. For harvesting records from a CSW, I would request a brief response (just getting IDs) then loop the list of IDs with GetRecordById requests with full responses. 10 records is is a default I think, you should be able to request more.
    – nmtoken
    Mar 22 '17 at 8:31
1

Update 19-3-2017:

I have now managed to harvest the CKAN API, by increasing the allowed timeout from 5 to 10 seconds. Still it takes quite some time to harvest all of the data, since most of the download occurs at speeds around 100 kb/s. Also, I do get a page with 998 results occasionally, although I correct for this by setting only increasing the 'start' parameter by 998 if this happens.

Things I tried to prevent the apparent throttling:

  • Included pauses of up to 10 seconds between requests
  • Using a different network: Tried this both on a glass-fiber connection (150/150), and a slower VDSL network (Both from the Netherlands)
  • Tried both with the api.data.gov API key as 'Authorization' in the request header, as well as without
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.