What are the data quality measures for open data?

Question

How does a consumer know they are getting good data? Are there standard frameworks for grading the quality of an open data set? Should there be metrics published around accuracy, completeness, timeliness or validity of the data? Should there be a minimum set of controls on the part of the publisher?

Answer 1

I think the question, as phrased, is impossible to answer well, but I will try.

Q: "How does a consumer know they are getting good data?"

A: Let me answer with more questions. How does a consumer know they are getting a good search result from Google? How do they know when the news is of high quality? It depends. As consumers get more interested and informed about something, they do better. The most savvy and informed consumers will compare a data set against a known source. Others have to rely on some degree of trust.

Q: "Are there standard frameworks for grading the quality of an open data set?"

A: In practice, there are defacto standards for metadata. For example, data.gov uses Dublin Core along with additional attributes. CKAN has many of the same attributes.

Also, for each type of data (or subfield) there are often industry standards or at least conventions. Good luck enumerating those!

A post from the Sunlight Foundation, Government Data Sets - Managing Expectations is a high-level gloss; it breaks down "dataset quality" into provenance, data quality, responsibility, maintenance, and documentation.

The above article is somewhat naive; the quality of a data set is not an independent thing. As Wikipedia - Data Quality points out, the quality of a data set depends on the question asked of it. There is no "one" measure of data quality. Rather, there is a subjective 'appropriateness' for each question you might ask of a data set. You can't ignore the subjective nature of data quality.

Q: "Should there be metrics published around accuracy, completeness, timeliness or validity of the data?"

A: There are advantages to doing so, sure, but there are costs too. Who provides the resources to do it? This question cannot be answered well in the abstract. This is a question of leadership and resources. If you want it, act. Advocate for it. Or do (hack, write, whatever) something for your city, country, state, province, or country.

Q: "Should there be a minimum set of controls on the part of the publisher?"

A: Maybe. This is complex.

1. Perhaps it is smart to release the data sets you have, regardless of quality, in whatever format you have available.
2. Perhaps later, over time, and perhaps with incentives, improve them and/or convert to better formats.
3. Increasing standards of data quality may act as barriers to publishing data; reducing your data inventory. This may be good or bad.
4. Some data releases may be criticized in any number of ways; for reasons in or out of your control.
5. Data may be used in ways that the government or collecting agency does not agree with.
6. Beware of publishing data that may be sensitive. It is difficult to anonymize data in the general case.

In summary, this is a hard problem with many pitfalls. Even well-meaning organizations may need significant prodding to make data releases happen.

Answer 2

I'm personally against attempting to measure concepts of 'data quality' because you can't assess the quality of data without discussing what you're attempting to do with it. We can discuss the quality of the data (and its associated metadata & documentation) for the purpose that it was originally collected, but 'bad' data might be fantastic for some other purpose.

As an example -- there are a lot of ground based solar observatories. Some days, the atmospheric conditions don't allow good seeing. (where 'seeing' has to do with how clearly they can resolve distant objects). The image might suck at the full resolution, but still be useful when reduced. And for someone trying to study atmospheric aberration, it could be fantastic data ... we just don't know.

That being said, last year the NSF ran a workshop to discuss data quality. As your comment gave me reason to finally get around to reading their report (but only skimmed through thus far), I'm going to take a few excepts from their conclusions starting on page 17 of their final report:

Conclusion

[...] we were able to distill these truths:

• We don't truly know what our data quality is today
• We need cooperative processes between creator, curator, and user
• Data curation should be as painless as possible

Major conclusions were:

Context

The chain from data capture through data curation to data users is too loose, and we need more and tighter interaction. Even defining "quality" without knowing the purpose of the data is difficult. Efficient capture of data including provenance and metadata is most easily done by working at the start of the process, not trying to retrofit quality in later. Later on, the aggregation of multiple databases often highlights errors that may have been overlooked in a single database, a problem aggravated by our lack of metrics for even separated areas.

Accounting

Few projects track their curation costs, and since many projects also do not measure the number and size of errors in their archive, we can not plan how much we should spend on quality assurance to achieve a given level of reliability. [...]

Technology

We lack toolkits for both quality management and workflow description. Different projects do not share expertise in essential activities such as auditing, provenance, and privacy policy. Tools are needed both for the actual data and for management of the metadata.

Selection

[...] We do not understand the tradeoff between more data and better data nor do we have a general model of tools to implement selection policies. [...]

Specializaton

Do different disciplines require different procedures? Mechanically collected sensor data has different errors than survey data, and databases involving people create privacy issues. Nevertheless there should be procedures that are shareable across domains. [...]

(disclaimer : I was invited to participate in the NSF workshop (as I was known to the organizers, who were from the LIS community), but when I realized they hadn't invited anyone from the NSIDC, who have done more work on the topic than I have, I ceded my seat to Ruth Duerr)

Answer 3

I think the best answer to this question is here:

http://answers.semanticweb.com/questions/1072/quality-indicators-for-linked-data-datasets

Open data and linked data are sometimes linked, but it's better to have the data both linked and open (Linked Open Data) than just open.

Answer 4

Tim Berners-Lee suggested the following 5 star approach:

Under the star scheme, you get one (big!) star if the information has been made public at all, even if it is a photo of a scan of a fax of a table -- if it has an open licence. The you get more stars as you make it progressively more powerful, easier for people to use.

★ Available on the web (whatever format) but with an open licence, to be Open Data

★★ Available as machine-readable structured data (e.g. excel instead of image scan of a table)

★★★ as (2) plus non-proprietary format (e.g. CSV instead of excel)

★★★★ All the above plus, Use open standards from W3C (RDF and SPARQL) to identify things, so that people can point at your stuff

★★★★★ All the above, plus: Link your data to other people’s data to provide context