Catalog.data.gov using CKAN API with python requests package

Question

I'm not sure if this is a requests issue, a CKAN API issue, or a catalog.data.gov issue.

I can do this GET query:

import requests
import json

r = requests.get('http://catalog.data.gov/api/search/dataset?q=sea_water_temperature')
f = r.json()
print f.keys()

[u'count', u'results']

a=f['results']

# print 1st item in list
print a[0]

meteorological-and-oceanographic-data-collected-from-the-national-data-buoy-center-coastal-mari

but when I try do the same query using POST, I get a different result:

d={'q':'sea_water_temperature'}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r2 = requests.post('http://catalog.data.gov/api/3/action/package_search', data=json.dumps(d), headers=headers)

f2 = r2.json()
print f2.keys()

[u'help', u'success', u'result']

print f2['result'].keys()

[u'count', u'search_facets', u'facets', u'results']

a2 =f2['result']['results']
print type(a2)
print a2[0]

<type 'list'>
{u'license_title': None, u'maintainer': None, u'relationships_as_object': [], u'private': False, u'maintainer_email': None, u'num_tags': 0, u'id': u'69dd90ef-82d9-4ae3-84ea-f8bc49e7de1d', u'metadata_created': u'2013-09-16T16:24:10.828099', u'metadata_modified': u'2013-11-14T15:09:10.161527', u'author': None, u'author_email': None, u'state': u'active', u'version': None, u'license_id': None, u'type': u'dataset', u'resources': [{u'resource_group_id': u'19a8fa83-675f-427d-a7ab-67fff586ab2b', u'cache_last_updated': None, u'revision_timestamp': u'2013-11-14T15:09:10.161527', u'webstore_last_updated': None, u'id': u'da37f12f-5029-4326-b238-2a4a5fe76473', u'size': None, u'state': u'active',....

What am I doing wrong?

Update: In response to @amercader's suggestion below, I tried using the API 3 for both the GET and PUT, but the GET didn't change, and the PUT still is not working. While the GET records returned seem correct, the PUT request records definitely are not datasets with "sea_water_temperature":

import requests
import json

r = requests.get('http://catalog.data.gov/api/3/search/dataset?q=sea_water_temperature')
f = r.json()
f.keys()

[u'count', u'results']

print f['count']
79873

f['results'][0:4]
[u'meteorological-and-oceanographic-data-collected-from-the-national-data-buoy-center-coastal-mari',
 u'meteorological-and-oceanographic-data-collected-from-the-national-data-buoy-center-coastal-mari41f45',
 u'meteorological-and-oceanographic-data-collected-from-the-national-data-buoy-center-coastal-mari46e38',
 u'meteorological-and-oceanographic-data-collected-from-the-national-data-buoy-center-coastal-mari4c218']

# now try same query with POST

d={'q':'sea_water_temperature'}
headers = {'Content-type': 'application/json'}
r2 = requests.post('http://catalog.data.gov/api/3/action/package_search', data=json.dumps(d), headers=headers)
f2 = r2.json()
print f2.keys()

[u'help', u'success', u'result']    In [11]:

print f2['result'].keys()

[u'count', u'search_facets', u'facets', u'results']

print f2['result']['count']
88049

for i in xrange(4):
    print f2['result']['results'][i]['title']

Parking lots owned and maintained by the U.S. Fish and Wildlife Service
LiDAR Derived Bare Earth Digital Elevation Model: Camas National Wildlife Refuge
LiDAR Boundary: Camas National Wildlife Refuge
USGS National Geologic Map Database Collection

I'm pretty sure there is something wrong with this POST call, right?

Answer 1

The POST syntax is fine - you get the same results using the web front end: http://catalog.data.gov/dataset?q=sea_water_temperature&sort=score+desc%2C+name+asc&ext_location=&ext_bbox=&ext_prev_extent=-254.53125%2C-75.84516854027044%2C54.84375%2C83.19489563661588

However the underscores are being treated as spaces, so effectively your query becomes: "sea" OR "water" OR "temperature". This is simply SOLR default syntax, and that has its pluses and minus. I think could successfully argue that the OR thing is not what you might expect, leading to all those results, but I'm not sure that the underscores as spaces is useful to users. (Can someone explain this aspect of SOLR?)

To get what you're after, try adding quote marks: http://catalog.data.gov/dataset?q=%22sea_water_temperature%22&sort=score+desc%2C+name+asc&ext_location=&ext_bbox=&ext_prev_extent=-254.53125%2C-79.43237075914709%2C54.84375%2C80.87282721505686

This gives 52 results, starting with ones with exactly this tag, which is what I think you're after.

If you're really just interested in this tag in particular, you can use just search this field using 'tags:' instead of 'q:' (web URL) or inside of the q value in the API.

You can save faff of using requests and JSON decoding by using ckanclient. Once you get the hang of it, it is dead easy:

>>> import ckanclient
>>> from pprint import pprint
>>> ckan = ckanclient.CkanClient('http://catalog.data.gov/api')
>>> pprint(ckan.action('package_search', q='tags:sea_water_temperature', rows=1))
{u'count': 51,
 u'facets': {},
 u'results': [{u'author': None,
               u'author_email': None,
               u'extras': [{u'key': u'bbox-east-long',
                            u'value': u'-94.867'},
                           {u'key': u'resource-type', u'value': u'dataset'},
                           {u'key': u'bbox-north-lat', u'value': u'8.085'},
...

Answer 2

It has nothing to do with GET vs POST, you seem to be using two different API versions. In the first case you are using version 1 of the CKAN API (http://catalog.data.gov/api/search/dataset), which is deprecated. This version returns a list of dataset ids.

In the second case you are using version 3 of the CKAN API (http://catalog.data.gov/api/3/action/package_search), which returns full dataset objects. This is the recommended version rigth now.

You can use GET or POST requests in both versions, although as I said, you will find version 3 more feature-rich.