Standards for documenting use caveats?

Question

One of the problems with distributing data is that people think they understand it, and try to use it in ways that may be incompatable with the files.

Examples:

• Years ago, one of the scientists I work with tried doing a longitudinal study of solar irradiance by summing up the total brightness for many years of calibrated images from a given telescope ... and found there was no variablity ... which was because the calibration normalized for total brightness across the image.
• The STEREO Beacon Data is tranmitted at reduced resolution and highly compressed ... but upscaled for movie-generation purposes. We replace the beacon images (the ones w/ '_n7eu' in the file names once the full-quality images are downlinked (they have '_n4eu' in the name), which means we have to deal with the 'compression artifact numbers' who insist they've found a UFO and that NASA's covering them up when we replace them with the better images.
• long wire antennas have different sensitivity than dual sphere antennas, and so what looks to be useful radio observations may actually be noise from the way in which the detector is contructed.
• more recent solar telescopes may be run in an exposure control mode in which the shutter will close early if there's a flare. Calibrated images that correct for the shorter exposure will have amplified noise that may ruin some types of analysis.
• Some disciplines don't distribute the error bars with the data iself, but as software calibration routines or as a peer-reviewed paper in the journal of record for their field.
• Event catalogs catalogs may be based on data with gaps in it (eg, the couple of times when SOHO was lost ; SDO eclipse season), so the catalog can't be used to assert that an event didn't happen.

... are there any standards for describing known biases of the data or other use caveats that may limit the applicability of the data? Or is anyone even aware of attempts to catalog the possible types of caveats & biases that may apply?

related : How can I trust the authenticity of an open data source? ; Standards for self-documenting text files?

Answer 1

I don't know that there can be a good standard other than what I was taught when studying environmental computer models, which is to be clear about everything up-front. Obviously this has to be documented somewhere in the expected place (like a README or the like) but the following things should probably be addressed on some way:

1. Methodology of how the data was collected.

2. Intended use, selection methodology

3. Any normalization which occurred prior to publication

4. Any known assumptions underlying the above three.

5. Any other notes that the collectors think might be useful.

This way people can read and check against their assumptions before they use the data in various ways.

Answer 2

In the old data.gov metadata schema, statistical datasets were required to supply a bunch of additional metadata that disclosed the following items:

• Statistical methodology
• Sampling
• Estimation
• Weighting
• Disclosure avoidance
• Questionnaire design
• Series breaks
• Non-response adjustment
• Seasonal adjustment
• Data quality (variances, CVs, CIs, etc)

This would generally seem to be good practice to follow for non-statistical datasets. Now, usually, these are just generally long documents that people need to read before they try to use and interpret such data.

There are initiatives out there, like UncertML (http://www.uncertml.org/) which are an attempt to carry these kinds of information directly with the data. Not sure how well it works, but they might be models from which to build for the kinds of scientific and engineering data you are asking about in your question.