8

How does a consumer know they are getting good data? Are there standard frameworks for grading the quality of an open data set? Should there be metrics published around accuracy, completeness, timeliness or validity of the data? Should there be a minimum set of controls on the part of the publisher?

  • 3
    Please split this into one question for each sentence you wrote above. Otherwise, an answer will need to be some unfortunate combination of overly opinionated, sprawling, or incomplete. – David J. May 24 '13 at 21:11
  • Asking this kind of question loaded with "should's" is problematic. There are pros and cons, actions and reactions, feedback loops, costs and benefits. – David J. May 24 '13 at 21:15
  • There are lies, d---ed lies, and statistics. If a data consumer trusts somebody, there should be an explicit expectation that part of this trust is misplaced. Unless one collects (and verifies) all the data him/herself, quality of the data is "undetermined". – Deer Hunter May 25 '13 at 11:39
  • This question/answer needs an update, see eg certificates.theodi.org – Ulrich Mar 31 '15 at 8:28
8

I think the question, as phrased, is impossible to answer well, but I will try.

Q: "How does a consumer know they are getting good data?"

A: Let me answer with more questions. How does a consumer know they are getting a good search result from Google? How do they know when the news is of high quality? It depends. As consumers get more interested and informed about something, they do better. The most savvy and informed consumers will compare a data set against a known source. Others have to rely on some degree of trust.

Q: "Are there standard frameworks for grading the quality of an open data set?"

A: In practice, there are defacto standards for metadata. For example, data.gov uses Dublin Core along with additional attributes. CKAN has many of the same attributes.

Also, for each type of data (or subfield) there are often industry standards or at least conventions. Good luck enumerating those!

A post from the Sunlight Foundation, Government Data Sets - Managing Expectations is a high-level gloss; it breaks down "dataset quality" into provenance, data quality, responsibility, maintenance, and documentation.

The above article is somewhat naive; the quality of a data set is not an independent thing. As Wikipedia - Data Quality points out, the quality of a data set depends on the question asked of it. There is no "one" measure of data quality. Rather, there is a subjective 'appropriateness' for each question you might ask of a data set. You can't ignore the subjective nature of data quality.

Q: "Should there be metrics published around accuracy, completeness, timeliness or validity of the data?"

A: There are advantages to doing so, sure, but there are costs too. Who provides the resources to do it? This question cannot be answered well in the abstract. This is a question of leadership and resources. If you want it, act. Advocate for it. Or do (hack, write, whatever) something for your city, country, state, province, or country.

Q: "Should there be a minimum set of controls on the part of the publisher?"

A: Maybe. This is complex.

  1. Perhaps it is smart to release the data sets you have, regardless of quality, in whatever format you have available.
  2. Perhaps later, over time, and perhaps with incentives, improve them and/or convert to better formats.
  3. Increasing standards of data quality may act as barriers to publishing data; reducing your data inventory. This may be good or bad.
  4. Some data releases may be criticized in any number of ways; for reasons in or out of your control.
  5. Data may be used in ways that the government or collecting agency does not agree with.
  6. Beware of publishing data that may be sensitive. It is difficult to anonymize data in the general case.

In summary, this is a hard problem with many pitfalls. Even well-meaning organizations may need significant prodding to make data releases happen.

  • Correction needed Data.gov (US) established its own metadata standard based on the W3C DCAT vocabulary in data.json v1.1. Which is used by all Federal Agency's to maintain their datasets on their websites as a mandated endpoint in agency.gov/data.json which is harvested by data.gov nightly project-open-data.cio.gov/v1.1/schema . Data.gov also harvests ISO and FGDC geospatial metadata collections and maps those standards to data.json – user33290 Aug 18 '16 at 7:26
5

I'm personally against attempting to measure concepts of 'data quality' because you can't assess the quality of data without discussing what you're attempting to do with it. We can discuss the quality of the data (and its associated metadata & documentation) for the purpose that it was originally collected, but 'bad' data might be fantastic for some other purpose.

As an example -- there are a lot of ground based solar observatories. Some days, the atmospheric conditions don't allow good seeing. (where 'seeing' has to do with how clearly they can resolve distant objects). The image might suck at the full resolution, but still be useful when reduced. And for someone trying to study atmospheric aberration, it could be fantastic data ... we just don't know.

That being said, last year the NSF ran a workshop to discuss data quality. As your comment gave me reason to finally get around to reading their report (but only skimmed through thus far), I'm going to take a few excepts from their conclusions starting on page 17 of their final report:

Conclusion

[...] we were able to distill these truths:

  • We don't truly know what our data quality is today
  • We need cooperative processes between creator, curator, and user
  • Data curation should be as painless as possible

Major conclusions were:

Context

The chain from data capture through data curation to data users is too loose, and we need more and tighter interaction. Even defining "quality" without knowing the purpose of the data is difficult. Efficient capture of data including provenance and metadata is most easily done by working at the start of the process, not trying to retrofit quality in later. Later on, the aggregation of multiple databases often highlights errors that may have been overlooked in a single database, a problem aggravated by our lack of metrics for even separated areas.

Accounting

Few projects track their curation costs, and since many projects also do not measure the number and size of errors in their archive, we can not plan how much we should spend on quality assurance to achieve a given level of reliability. [...]

Technology

We lack toolkits for both quality management and workflow description. Different projects do not share expertise in essential activities such as auditing, provenance, and privacy policy. Tools are needed both for the actual data and for management of the metadata.

Selection

[...] We do not understand the tradeoff between more data and better data nor do we have a general model of tools to implement selection policies. [...]

Specializaton

Do different disciplines require different procedures? Mechanically collected sensor data has different errors than survey data, and databases involving people create privacy issues. Nevertheless there should be procedures that are shareable across domains. [...]

(disclaimer : I was invited to participate in the NSF workshop (as I was known to the organizers, who were from the LIS community), but when I realized they hadn't invited anyone from the NSIDC, who have done more work on the topic than I have, I ceded my seat to Ruth Duerr)

5

I think the best answer to this question is here:

http://answers.semanticweb.com/questions/1072/quality-indicators-for-linked-data-datasets

Open data and linked data are sometimes linked, but it's better to have the data both linked and open (Linked Open Data) than just open.

  • Re: "Open data and linked data are inextricably linked". No. Linked Open Data advocates may say that everything is linked to Linked Open Data. Others are less likely to say this. Many people that consume data and visualize it on a daily basis may not care or may prefer other formats. – David J. May 24 '13 at 21:56
2

Tim Berners-Lee suggested the following 5 star approach:

Under the star scheme, you get one (big!) star if the information has been made public at all, even if it is a photo of a scan of a fax of a table -- if it has an open licence. The you get more stars as you make it progressively more powerful, easier for people to use.

★ Available on the web (whatever format) but with an open licence, to be Open Data

★★ Available as machine-readable structured data (e.g. excel instead of image scan of a table)

★★★ as (2) plus non-proprietary format (e.g. CSV instead of excel)

★★★★ All the above plus, Use open standards from W3C (RDF and SPARQL) to identify things, so that people can point at your stuff

★★★★★ All the above, plus: Link your data to other people’s data to provide context

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.