Standards for documenting gaps in data?

Question

Often, we have catalogs of events or features that are based off of a time-series of data that has gaps.

I was wondering if there were any standards for describing those gaps, and what type of gap it might be, eg:

• there will never be data for this period. (data wasn't downlinked; sensor failed and was replaced, etc.)
• there's data, but access is restricted. (eg, data didn't meet quality standards for the intended purpose, and so isn't distributed w/out talking to the PI directly)
• the data has been 'lost'. (eg, authoritative archive no longer has a copy; as it existed, someone might have it but it'd be suspect unless chain of custody could be proven)

related : Standards for documenting use caveats?

update : I guess I should describe my particular use cases -- I'm not actually dealing with your typical 1-dimensional data, where you might have a file w/ a month's worth of observations in it, and so gaps within the file or the use of 'fill values' are obvious. I have solar telescope data, and we have periods where the telescopes perform calibration routines (eg, close the shutters, take a picture of a calibration lamp, so it wasn't observing the sun), some spacecraft have orbits where they go through an 'eclipse' where their orbit takes them behind the earth. Others have 'bake out' periods where they heat the CCD to improve its sensitivity). For the most part, our data is cataloged by the discrete images, and not as a series, so although I can identify gaps larger than a given time, I don't necessarily have documentation of why those gaps exist.

There's a SOHO/EIT bakeout webpage, but I suspect it's out of date (one in 2012 is listed 'TBD'), and it doesn't list the gap when they lost but recovered the spacecraft. But for catalogs and other higher-level products created using that data, the community knows about that caveat ... anyone new is going to have no idea as it's not explicitly mentioned. Take the CDAW LASCO CME catalog ... there's a gap for 3 months in 1998 ... but no explanation why, or specifically mentioning when the gap started (June 25th to ... depends on the instrument ... between 25 sept and 5 nov).

If standards exist, I'd also like to use them when returning results from the Virtual Solar Observatory ... you searched for a period that included a gap, I return a record that says 'there was a gap from (start) to (end) of type (whatever)', rather than a generic 'no records found'. (especially as in the case of recent observations, it might be a transient 'try back later, it's not processed yet')

Answer 1

I do not know of standards in this area, but I do know that many data owners document missing data or other known issues in a dataset. This is generally documented in either the site from which the dataset is linked or in the metadata of the dataset itself.

Answer 2

I think the key is to that the data gathering and usage document should be pretty heavily integrated. This really should be documented as part of intended use and use caveats. In other words, documenting data sampling methods and assumptions, and problems gathering or releasing the data is what is needed, and gaps in data are just a part of that picture.

I dont think these need to be separately documented. Rather it is important to provide users an accurate picture of which this is a piece.

Answer 3

In RDF, you can do the trick mixing these ingredients:

• an observation ontology or vocabulary;
• RDF blank nodes;
• a provenance ontology.

Sensor and Observations Ontologies

There are many ontologies for measurements and observations, see e. g. this (outdated) list.
In the example below, I will use quick and dirty ad-hoc vocabulary, do not take it seriously.

RDF blank nodes

Think of blank nodes as anonymous resources or existential variables. They are similar to marked nulls from the article I have pointed to in the comment above. For example, Wikidata t-values are in fact blank nodes.

Provenance ontology

The most prominent provenance ontology is The PROV ontology.
I hope the prov:wasInvalidatedBy property will be suitable for your needs.

Example

# Prefixes
@prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
@prefix xsd:  <http://www.w3.org/2001/XMLSchema#> .
@prefix prov: <http://www.w3.org/ns/prov#> .
@prefix ex:   <http://example.org#> .

# T-Box

ex:Observation rdfs:subClassOf prov:Entity ;
               rdfs:label "Observation" .
ex:time        rdf:type rdf:Property ;
               rdfs:label "Observation exact time@en" ;
               rdfs:domain ex:Interval ;
               rdfs:range  xsd:dateTime .
ex:value       rdf:type rdf:Property ;
               rdfs:label "Observation result (in km/s)" ;
               rdfs:domain ex:Interval ;
               rdfs:range  xsd:dateTime .
ex:interval    rdf:type rdf:Property ;
               rdfs:label "Observation interval"@en ;
               rdfs:domain ex:Observation ;
               rdfs:range ex:Interval ;
ex:Interval    rdf:type rdfs:Class ;
               rdfs:label "Observation interval"@en ;
               rdfs:description "Observation should be
               performed at least once during an interval"@en .
ex:start       rdfs:subPropertyOf ex:Property ;
               rdfs:label "Interval start date"@en ;
               rdfs:domain ex:Interval ;
               rdfs:range  xsd:date .
ex:end         rdf:type rdf:Property ;
               rdfs:range ex:Interval ;
               rdfs:domain "Interval end date"@en ;
               rdfs:range  xsd:date .      

# A-Box

# Successfull observation
ex:observation1 rdf:type ex:Observation .
                ex:interval  [ rdf:type ex:Interval ;
                               ex:start "2012-09-01"^^xsd:date ;
                               ex:end "2012-09-05"^^xsd:date ] ;
                ex:time "2012-09-04T20:00:00Z"^^xsd:dateTime ; 
                ex:value "150.5"^^xsd:float ;
                prov:wasGeneratedBy  ex:ourMainActivity .

# Successfull observation, but the result was lost
ex:observation2 rdf:type ex:Observation .
                ex:interval  [ rdf:type ex:Interval ;
                               ex:start "2012-09-06"^^xsd:date ;
                               ex:end "2012-09-10"^^xsd:date ] ;
                ex:time "2012-09-04T20:00:00Z"^^xsd:dateTime ; 
                ex:value _:b0 ;
                prov:wasGeneratedBy  ex:ourMainActivity ;
                prov:wasInvalidatedBy ex:hard_disk_failure .

# Observation was not performed due to unknown reasons
ex:observation3 rdf:type ex:Observation ;
                ex:interval  [ rdf:type ex:Interval ;
                               ex:start "2012-09-11"^^xsd:date ;
                               ex:end "2012-09-16"^^xsd:date ] ;
                ex:time _:b1 ; 
                ex:value _:b2 ;
                prov:wasInvalidatedBy _:b3 .

ex:ourMainActivity rdf:type prov:Activity ;
                   prov:startedAtTime  "2012-04-25T01:30:00Z"^^xsd:dateTime .

ex:hard_disk_failure rdf:type prov:Activity ;
                     rdfs:label "Hard disk failure" ;
                     prov:endedAtTime "2012-09-11T01:31:00Z"^^xsd:dateTime .

---

If gaps cover the whole periods, you could assert that the whole dataset was invalidated (this dataset may be in CSV format):

ex:dataset_1 rdf:type dcat:Dataset, prov:Entity ;
             dct:temporal <http://reference.data.gov.uk/id/quarter/2006-Q1> ;
             dcat:distribution  [ rdf:type dcat:Distribution ;
                                  dcat:mediaType "text/csv" ;
                                  dcat:downloadURL [] ] ;
             prov:wasInvalidatedBy ex:hard_disk_failure .

ex:hard_disk_failure rdf:type prov:Activity ;
                     rdfs:label "Hard disk failure" ;
                     prov:endedAtTime "2012-09-11T01:31:00Z"^^xsd:dateTime .