Displaying Trust and Data Provenance?

Question

What would be the most useful way to display data provenance on an open data site? Overarching statements on the site seem to be limited in that amongst many datasets, at least some will likely have data gaps or problems. In other cases, an open data site might be displaying data from many sources, with varying quality or trust.

Is putting provenance information in the metadata of each dataset sufficient or would some graphical notation (e.g., Sir Tim Berners-Lee's 5 star data) be helpful? If either is true, which aspects of provenance are most important to validate?

Notes:

• The W3C provides a description of provenance.
• @Igelkott asked about how to validate the authenticity of open data, but this question is focused on how to clearly display verified data.
• Disclaimer: I am the Evangelist for Data.gov.

Answer 1

My issue would be what the purpose of displaying the provenance is.

As I've suggested in some of my questions on here, some of my concerns are about tracing the issues that might be in the data, and sometimes you have to go back and look to see how it's been processed and what it's derived from to tell what the possible issues might be.

(eg I've run into at least once case where the folks calibrating the data hadn't considered big vs. little endian when they converted from 68000 to PPC processors ... the issue wasn't caught 'til they went from PPC to intel and realized that all of the PPC processed data (years worth) was defective. ... but architecture the processing was run on isn't always captured when people talk about provenance)

Personally, I'm of the opinion that for the sake of open data, you shouldn't just share the provenance of the data set in question, but you should try to describe that data's relationship with all of the rest of the data that you serve. (rule #4 in "5 star data) After all, it might be that the data that someone found has some more processed form that would be better for their needs ... or formatted / packaged differently. ... and you're not going to get that from only tracking provenance, as that only goes in one direction.

(disclaimer : years ago, I gave a talk at the AGU on the need for a model to discuss relationships between data (warning: 18MB PPT file) ... unfortunately, I've gotten bogged down with other stuff for the past few years ... but the DataCite schema has RelatedIdentifier and a decent list for relationType to get people started)

Answer 2

I'll take a stab... On the web, there's only one sure way to display something related to a piece of text/number/whatever - hyperlinking (yes, there are tooltips, but they aren't likely to be discoverable or accessible). If you are using HTML tables, linked notes are OK.

Ideally, users should be able to trace the origins (and the full chain of transformations) of the data at each level where it matters: overall provenance for the dataset, links at each column's title when there were unusual/less than obvious transformations/normalizations applied, and sources for particular items when expert judgment or multiple-source fusion were used instead of solid data from one source.

The crucial step is enabling users to see if there were gaps/various kinds of missing data in the original dataset and how these gaps were filled.

Additionally, it may be very nice to have access to historical revisions of the data.

Answer 3

I'd suggest that the most important thing is that, where possible, you are clear about several different factors. Many of these are tied up with the definition of provenance:

• Who is publishing the data? The organisation doing the publisher might not be directly responsible for its collection and maintenance (e.g. govt data being published in a central or third-party portal)

• How was the data collected? There are many ways that data might be collected, from detailed or ad hoc surveys, applications, crowd-sourcing, statistical analysis, etc. Identifying the method of collection might help identify potential gaps, biases, etc in the data.

• How is the data maintained? This relates to ongoing data quality management. Does the data get updated? And if so, how does that happen

Its difficult to boil this down into a simple set of display criteria, e.g. a 5-star scheme, as there are many different dimensions.

The best approach would be to ensure that there these different factors are clearly documented and that the documentation is clearly referenced from a dataset homepage.

Machine-readable descriptions, e.g. http://www.w3.org/TR/prov-overview/, might be helpful too.

Answer 4

My company, Bitscover by name, is developing a trust rating system for the data that we collect from private sector sources. We don't yet feel confident enough in our "navigation" of government data sources to do this for public sector data, but we are always open to "learning enough" to do this.

Our private sector rating is a composite rating based on two main factors (and other, smaller ones). They are 1) the source or "provenance" of the data and 2) the apparent quality of the data.

1) Provenance: All other things being equal, a source like the New York Times or Wall St. Journal is considered more reliable than say, the local newspaper.

2) Number 1) notwithstanding, a well-reasoned, well-written article in the local newspaper carries more weight than a random "thrown together" article from the Times or the Journal.

3) This is technically under 2), but I'm going to present it separately. If the author of the "local" paper's article was the Editor-in-Chief, and the author of the Times or Journal article was a new, junior staffer, or "no name" writer, that would, of course weigh in favor of the local paper's piece.

So you have a multi factor analysis that figures into a "composite" score.