The main requirements for our open data directory are:

  1. XML-based by default with ability to switch to JSON. It's important the easy way to make it human readable with just linking to XSL.
  2. All the data must be reachable by robot from the single starting point (root) without using interfaces like SPARQL. Something like the list of categories or entity types.
  3. Index of huge data collection of homogeneous entities must partitioned in some way (paged, by date period etc.). It must be obvious to robot that data are partitioned.
  4. Only reference to large entity can present in collection index, but not whole entity description. I.e. there should be a marker in the index unambiguously saying that data in index is not full and you have to follow a link to get more.

What formats and schemas should I use? What vocabularies have support for such indexes and data partitioning? Are there public projects with such solutions that I could explore and use as examples?

4 Answers 4


I would suggest:

When dereferencing the root URL, point to a metadata document (via RDFa and/or conneg), say http://yoursite.com/meta. When dereferencing this document, provide a description of datasets using DCAT. every URI there, when dereferenced can show metadata (file size, creation dates, etc) and include a dcat:downloadURl link with the actual data. I'm not sure about the paging issue, but all of the above can be done in say, apache + flat files. Of course there are other more sophisticated solutions but I think this pretty much covers most of what you ask for.

  • Thanks, DCAT looks as a good candidate for starting point vocabulary. May 29, 2013 at 10:05

The Linked Data API can help here. It's an open specification that defines how to expose data in a SPARQL endpoint as Linked Data, including descriptions of resources and lists of resources.

The API provides a number of features that meet you needs:

  • Configurable list of entities, e.g. by type or property, with support for paging, sorting and filtering
  • Configurable level of descriptions, e.g. a default description might be a simple label + type, a complete description might include all properties
  • Data exposed in simple XML and JSON formats, as well as RDF serializations
  • RESTful design making it easy to crawl the paged collections of data

More background:

The Linked Data API was designed to help support publishing of Government Linked Data in the UK. Its been used to publish data from the Environment Agency. You could refer to that as an example

  • Thank you for the answer. LDA is quite interesting. Unfortunately it lacks single starting points that limits discoverability: you have to list all endpoints somewhere that will require manual handling by users. May 28, 2013 at 12:43
  • 2
    The endpoints themselves can be referred to by a starting document.
    – ldodds
    May 28, 2013 at 13:30
  • Your suggestion seems reasonable and I hope to find an example of such starting endpoint. May 29, 2013 at 9:45

Going back to your original requirements for the linked data directory--is that as thorough as you need? Or is that the minimum? We're looking at the the best way to collect data to get it ready for linked directory. Currently:

Five Rules:

  1. Structured: Give data users the ability to re-index, reaggregate, and remix your data points.
  2. Machine Readable: Good: CSV, XML, JSON. Bad: PDF, TIFF, JPG
  3. Documentation: How did you collect it? What assumptions need to be explained? What do column names mean?
  4. Open Data Formats: Proprietary data formats put up unnecessary barriers to adoption. Simple, open data formats let anybody have access.
  5. Design for Open APIs: REST APIs are like websites full of data. Users can simply reload to get the latest, up-to-date versions.

You might want to look at CKAN as a starting point.

  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. Jun 1, 2013 at 7:20
  • It's a software package. If the link goes away, the software goes away and the answer is no longer relevant. Feel free to delete my answer when that happens. Given that the U.S. government is using it for their data portal, that seems unlikely to me.
    – Tom Morris
    Jun 1, 2013 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.