I'm trying to harvest data from the CKAN API of Data.gov, using Python code. Doing this however, I ran into problems. I use the following URL:
http://catalog.data.gov/api/3/action/package_search?rows=1000&start=0
And increment the 'start' parameter after every request. I have run into two difficulties:
- It seems like I get throttled: Sometimes after a few pages or even at the first page, and other times only after 120 pages (1000 results each) I experience request timeouts, and progress becomes extremely slow (e.g. 100 kb/s). I do have an apiKey for api.data.gov (Is this actually valid for the CKAN API?), but if I include this as 'Authorization' in the http header, still timeouts occur.
- Using an offset of 42000, often only 998 packages are returned, although it is still far from the end of the list (192.633). This occurs by doing the following request: http://catalog.data.gov/api/3/action/package_search?rows=1000&start=42000. Other times it actually does return 1000 results here, but returns 998 results on a later page...
Can I do anything on my end to prevent the throttling from occuring, or is something else wrong?
https://catalog.data.gov/csw?service=CSW&version=2.0.2&request=GetCapabilities
only works sometimes, now and then returns an xml exception-report for me:Could not load repository (local): (OperationalError) SSL SYSCALL error: EOF detected "\n SELECT c.oid\n FROM pg_catalog.pg_class c\n LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace\n WHERE (pg_catalog.pg_table_is_visible(c.oid))\n AND c.relname = %(table_name)s AND c.relkind in ('r','v')\n " {'table_name': u'records'}
– Tom Brouwer Mar 19 '17 at 14:09