Best practices for huge explorable linked data directories

Question

The main requirements for our open data directory are:

1. XML-based by default with ability to switch to JSON. It's important the easy way to make it human readable with just linking to XSL.
2. All the data must be reachable by robot from the single starting point (root) without using interfaces like SPARQL. Something like the list of categories or entity types.
3. Index of huge data collection of homogeneous entities must partitioned in some way (paged, by date period etc.). It must be obvious to robot that data are partitioned.
4. Only reference to large entity can present in collection index, but not whole entity description. I.e. there should be a marker in the index unambiguously saying that data in index is not full and you have to follow a link to get more.

What formats and schemas should I use? What vocabularies have support for such indexes and data partitioning? Are there public projects with such solutions that I could explore and use as examples?

Answer 1

I would suggest:

When dereferencing the root URL, point to a metadata document (via RDFa and/or conneg), say http://yoursite.com/meta. When dereferencing this document, provide a description of datasets using DCAT. every URI there, when dereferenced can show metadata (file size, creation dates, etc) and include a dcat:downloadURl link with the actual data. I'm not sure about the paging issue, but all of the above can be done in say, apache + flat files. Of course there are other more sophisticated solutions but I think this pretty much covers most of what you ask for.

Answer 2

The Linked Data API can help here. It's an open specification that defines how to expose data in a SPARQL endpoint as Linked Data, including descriptions of resources and lists of resources.

The API provides a number of features that meet you needs:

• Configurable list of entities, e.g. by type or property, with support for paging, sorting and filtering
• Configurable level of descriptions, e.g. a default description might be a simple label + type, a complete description might include all properties
• Data exposed in simple XML and JSON formats, as well as RDF serializations
• RESTful design making it easy to crawl the paged collections of data

More background:

• General overview
• Specification
• Elda is an Open Source implementation

The Linked Data API was designed to help support publishing of Government Linked Data in the UK. Its been used to publish data from the Environment Agency. You could refer to that as an example

Answer 3

Going back to your original requirements for the linked data directory--is that as thorough as you need? Or is that the minimum? We're looking at the the best way to collect data to get it ready for linked directory. Currently:

Five Rules:

1. Structured: Give data users the ability to re-index, reaggregate, and remix your data points.
2. Machine Readable: Good: CSV, XML, JSON. Bad: PDF, TIFF, JPG
3. Documentation: How did you collect it? What assumptions need to be explained? What do column names mean?
4. Open Data Formats: Proprietary data formats put up unnecessary barriers to adoption. Simple, open data formats let anybody have access.
5. Design for Open APIs: REST APIs are like websites full of data. Users can simply reload to get the latest, up-to-date versions.

Answer 4

You might want to look at CKAN as a starting point.