Restrict search to open datasets on CKAN's Data Hub?

Question

The Data Hub, powered by CKAN, is an impressive collection of datasets available on the web. Currently it lists more than 6000 datasets — however, not all of them are available under an Open Data license.

Is there any way to only search for or list open datasets that are compatible with the Open Definition?

Answer 1

There is a simple answer thanks to the isopen parameter. Here's the query:

http://datahub.io/api/3/action/package_search?q=isopen:true

Note that q is basically the classic solr query paramemter so if you want to combine this with normal queries you do stuff like:

http://datahub.io/api/3/action/package_search?q=gold%20isopen:true

Or, slightly more elegantly:

http://datahub.io/api/3/action/package_search?fq=isopen:true&q=gold

Note that this just checks whether the license is open (as in conforms to the Open Definition).

Strictly, for full openness as per the open definition the data should be accessible (meaning available (in bulk) and machine-readable). As this is difficult to check automatedly these requirements are not part of the isopen computation in CKAN at the moment.

Answer 2

From poking around at http://demo.ckan.org/dataset?_license_id_limit=0 I see that the left rail allows you to filter by license ID. I can't find any syntax for specifying a license ID as part of the query.

I don't know why that isn't exposed on the DataHub.

There's a closed ticket from four years ago that indicates that a feature was added to search only open data sets, and the demo CKAN API exposes an "is_open" boolean value... but none of it seems to be very well exposed to web users.

Answer 3

I can't find a way in the web interface to do it and the API appears to have incomplete data. I was able to download a dump and parse through it and create a CSV with the file name and the license URL.

Here is the script [Github] I threw together**

import csv
import json
import os

import rdfxml

class Sink(object): 
   def __init__(self): self.result = []
   def triple(self, s, p, o): self.result.append((s, p, o))

def rdfToPython(s, base=None): 
   sink = Sink()
   return rdfxml.parseRDF(s, base=None, sink=sink).result

PATH = r"C:\Users\John\Desktop\ckan-rdf"

csv_file = open("licenses.csv", 'w')
writer = csv.writer(csv_file)

for file_name in os.listdir(PATH):
    with open(os.path.join(PATH, file_name)) as f: 
        s_rdf = f.read()
        try:
            pyth = rdfToPython(s_rdf)
        except:
            print "Error Parsing: " + file_name
            continue
        for t in pyth:
            if t[1] == u"<http://purl.org/dc/terms/rights>":
                licsn = t[2][1:-1]
                writer.writerow([file_name, licsn])

csv_file.close()

** I don't consider this to be a complete solution as the library I used threw several errors when parsing the RDF files, unless you have a good understanding of the licenses the URLs are almost useless, and my knowledge of the RDF format is very poor (I probably missed things)

---

You can download the CSV file by clicking this button in the upper left hand corner.