I work at a University which is trying to open up some of its data. We make some of our data available as PDF or Excel documents.

The 5-star deployment scheme has been referenced a fair bit in Europe and I'm beginning to meet the question of "how do we get up to 5-star level?" (with perhaps the spoken or unspoken addition 'as easily and as cheaply as possible').

To give some examples we have PDF/Excel downloads of Forestry Statistics, and some Historical data (Swedish page).

I note we don't currently provide the data in CSV, but we could for example take a simple Excel table as an example of our "2-star" data:

http://www.slu.se/Documents/externwebben/webbtjanster/statistik-om-skog/Tidsserier/1923-nu/Agoslag.xlsx

I've posted a CSV file to Google Spreadsheets: https://docs.google.com/spreadsheets/d/1iA3-3XG121q4uE9_2Kh8z1Fbx-RN8TjYXY8zroq7AGA/edit?usp=sharing

I'm not very clear on RDF concepts, but am learning, whereas the people I'm talking to have even less technical knowledge.

What needs to be done to move from, say a 3-star level where a simple table is made available in a CSV file up to levels 4 and 5?

Is the addition of some sort of metadata file enough, or does RDF metadata need to be added to the CSV file?

up vote 4 down vote accepted

The current situation of your data

  • Data tables in PDF (example from your forestry statistics page) are a great example of 1-star data: The data is on the web, but it's pretty much unusable for computer programs without some heavy-duty data extraction. After all, PDF is where data goes to die. My advice: Just don't do it.

  • Your example Excel file is actually a good start. It's 2-star data, since it's still in a proprietary file format, but it's structured and machine-readable. It could easily be turned into 3-star data ("Save as CSV"). In other words, the difference between 2-star and 3-star data is mainly ideological and much less technical.

Getting to 3-star data is not a problem. Turning it into 4-star and 5-star data is where things start to get interesting.

The structure of your example data

  • Your data has three dimensions: County (expressed as number, code, and abbreviation), land use class, and year.
  • For each combination of these three dimensions, there is one data value, which is the area in hectare.

Expressing two dimensions in tabular form — such as an Excel spreadsheet — is trivial: On dimension (e.g. county) goes on the vertical axis, the other dimension (e.g. land use class) on the horizontal axis. However once the dimensionality is larger than 2, it becomes tricky to represent the data in two dimensions (e.g. in a table on a computer screen).

There are standard solutions to deal with multi-dimensional data, such das Pivot Tables and OLAP Cubes. In the context of Linked Data, the RDF Data Cube Vocabulary is the current best practice for expressing multi-dimensional data.

What needs to be done to move to 4- or 5-star data?

4-star data means that you "use open standards from W3C" (such as RDF) and/or "use URIs to denote things" so that people can point at your stuff. 5-star data means that you link your data to other people's data.

What does that mean in case of your statistical data? Well, basically it means that you would have to turn your statistical data into RDF Data Cubes. With the current state of tool support, this still requires a lot of effort and expertise. While RDF Data Cubes are a great thing in theory, so far they are not really relevant in practice because, outside of the Semantic Web research community, not that many people use this technology.

Other things you could do

If you do not want to go for RDF Data Cubes — and unless you have strong reasons in favor, I would not recommend it — there are other things you could do to make your data more machine-readable:

Publish your data as (compressed) flat CSV tables

This makes it trivial to

  • import the data into a spreadsheet application and create a pivot table
  • import the data into OpenRefine and perform advanced operations
  • turn the data into RDF Data Cubes, if someone wants to do it and already has the necessary skills

You could even add another column containing URIs representing the counties (e.g. https://www.wikidata.org/entity/Q104926 for Uppsala County). This would fulfil the requirements for 5-star data (although we skipped the requirements for 4-star data).

Keep an eye on the Metadata Vocabulary for Tabular Data

So far it's only a W3C Working Draft, but it sounds promising:

Validation, conversion, display and search of tabular data on the web requires additional metadata that describes how the data should be interpreted. This document defines a vocabulary for metadata that annotates tabular data. This can be used to provide metadata at various levels, from collections of data and how they relate to each other down to individual cells within a table.

  • 3
    +1 for the remarks on PDFs. PDF 'data' is not 'open' at all IMO. – Jan Doggen Mar 5 '15 at 14:17
  • 1
    @Patrick Good stuff, I'll have to read through this more carefully. Well-presented and good links. – Dave Alger Mar 7 '15 at 9:35
  • @JanDoggen PDF is going to cause headaches in the future, sure, but getting things from paper and into downloadable form is an important step for many who have been producing "open" data since before there were computers. It's what I like about the star model, you give a big fat star for putting it online and then say "why not make it more open?" – Dave Alger Mar 7 '15 at 9:39

"What needs to be done to move from, say a 3-star level where a simple table is made available in a CSV file up to levels 4 and 5?"

To jump from level 3 to level 4, the csv data needs to be converted to RDF. You could display it in an html document, and mark it up with RDFa. You could also serialize it to JSON-LD or Turtle.

If you view the example for four star data:
http://5stardata.info/gtd-4.html
you can see the RDFa written into the markup.
you don't need to alter your csv.

this is a great question. if you can post your csv here, lets markup it up to level four and five, and keep it here for others to see

  • That sounds like a good idea. I don't have any CSV data to hand, but taking data closer to "home" for me we have some Excel documents that can be quite simple and I should think can simply be saved as CSV for discussion purposes. Let me know. – Dave Alger Mar 5 '15 at 8:55
  • if you want to upload them to google spreadsheets and share we can go from there. – albert Mar 5 '15 at 14:20
  • I added a link to a google spreadsheet. – Dave Alger Mar 7 '15 at 9:25
  • i don't understand the columns. sorry. i can see the data is laid out by year but i'm stumped past that. if you can clarify some of the meanings, i would be of more help – albert Mar 7 '15 at 20:38
  • I'm not sure what you mean by "meanings" of columns. They're given in both Swedish and English (rows 6+7). There are technical land-use classes given in column D - things like Forest Land, Marshland, Mountain etc. The county names are fairly typical short forms: "Stockholm" is "Sthlm", there's an associated Swedish county code. Not quite sure what you're asking. – Dave Alger Mar 9 '15 at 12:30

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.