Often, we have catalogs of events or features that are based off of a time-series of data that has gaps.
I was wondering if there were any standards for describing those gaps, and what type of gap it might be, eg:
- there will never be data for this period. (data wasn't downlinked; sensor failed and was replaced, etc.)
- there's data, but access is restricted. (eg, data didn't meet quality standards for the intended purpose, and so isn't distributed w/out talking to the PI directly)
- the data has been 'lost'. (eg, authoritative archive no longer has a copy; as it existed, someone might have it but it'd be suspect unless chain of custody could be proven)
related : Standards for documenting use caveats?
update : I guess I should describe my particular use cases -- I'm not actually dealing with your typical 1-dimensional data, where you might have a file w/ a month's worth of observations in it, and so gaps within the file or the use of 'fill values' are obvious. I have solar telescope data, and we have periods where the telescopes perform calibration routines (eg, close the shutters, take a picture of a calibration lamp, so it wasn't observing the sun), some spacecraft have orbits where they go through an 'eclipse' where their orbit takes them behind the earth. Others have 'bake out' periods where they heat the CCD to improve its sensitivity). For the most part, our data is cataloged by the discrete images, and not as a series, so although I can identify gaps larger than a given time, I don't necessarily have documentation of why those gaps exist.
There's a SOHO/EIT bakeout webpage, but I suspect it's out of date (one in 2012 is listed 'TBD'), and it doesn't list the gap when they lost but recovered the spacecraft. But for catalogs and other higher-level products created using that data, the community knows about that caveat ... anyone new is going to have no idea as it's not explicitly mentioned. Take the CDAW LASCO CME catalog ... there's a gap for 3 months in 1998 ... but no explanation why, or specifically mentioning when the gap started (June 25th to ... depends on the instrument ... between 25 sept and 5 nov).
If standards exist, I'd also like to use them when returning results from the Virtual Solar Observatory ... you searched for a period that included a gap, I return a record that says 'there was a gap from (start) to (end) of type (whatever)', rather than a generic 'no records found'. (especially as in the case of recent observations, it might be a transient 'try back later, it's not processed yet')