6

The Data Hub, powered by CKAN, is an impressive collection of datasets available on the web. Currently it lists more than 6000 datasets — however, not all of them are available under an Open Data license.

Is there any way to only search for or list open datasets that are compatible with the Open Definition?

5

There is a simple answer thanks to the isopen parameter. Here's the query:

http://datahub.io/api/3/action/package_search?q=isopen:true

Note that q is basically the classic solr query paramemter so if you want to combine this with normal queries you do stuff like:

http://datahub.io/api/3/action/package_search?q=gold%20isopen:true

Or, slightly more elegantly:

http://datahub.io/api/3/action/package_search?fq=isopen:true&q=gold

Note that this just checks whether the license is open (as in conforms to the Open Definition).

Strictly, for full openness as per the open definition the data should be accessible (meaning available (in bulk) and machine-readable). As this is difficult to check automatedly these requirements are not part of the isopen computation in CKAN at the moment.

2

From poking around at http://demo.ckan.org/dataset?_license_id_limit=0 I see that the left rail allows you to filter by license ID. I can't find any syntax for specifying a license ID as part of the query.

I don't know why that isn't exposed on the DataHub.

There's a closed ticket from four years ago that indicates that a feature was added to search only open data sets, and the demo CKAN API exposes an "is_open" boolean value... but none of it seems to be very well exposed to web users.

  • When I was messing with the API the package list function seems to be missing quite a bit of data. Would you be so kind as to provide a link to the ticket? – John May 12 '13 at 18:09
  • 2
    DataHub is only running CKAN 1.8 and it is a shame it is not kept up to date. demo.ckan.org is CKAN 2.0 which has the license search facet. I suspect the old ticket mentioned was a reference to the filter_by_openness search option in the API, but it was not carried through to SOLR which came in SOLR 1.1. – D Read May 13 '13 at 12:58
  • @johnthexiii sorry. I've edited the answer to link to the ticket. – Joe Germuska May 13 '13 at 16:47
1

I can't find a way in the web interface to do it and the API appears to have incomplete data. I was able to download a dump and parse through it and create a CSV with the file name and the license URL.

Here is the script [Github] I threw together**

import csv
import json
import os

import rdfxml

class Sink(object): 
   def __init__(self): self.result = []
   def triple(self, s, p, o): self.result.append((s, p, o))

def rdfToPython(s, base=None): 
   sink = Sink()
   return rdfxml.parseRDF(s, base=None, sink=sink).result

PATH = r"C:\Users\John\Desktop\ckan-rdf"

csv_file = open("licenses.csv", 'w')
writer = csv.writer(csv_file)

for file_name in os.listdir(PATH):
    with open(os.path.join(PATH, file_name)) as f: 
        s_rdf = f.read()
        try:
            pyth = rdfToPython(s_rdf)
        except:
            print "Error Parsing: " + file_name
            continue
        for t in pyth:
            if t[1] == u"<http://purl.org/dc/terms/rights>":
                licsn = t[2][1:-1]
                writer.writerow([file_name, licsn])

csv_file.close()

** I don't consider this to be a complete solution as the library I used threw several errors when parsing the RDF files, unless you have a good understanding of the licenses the URLs are almost useless, and my knowledge of the RDF format is very poor (I probably missed things)


You can download the CSV file by clicking this button in the upper left hand corner.

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.