Standards for self-documenting text files?

Question

One of the advantages of most scientific file formats is that you have a block of attached metadata that tells a reader how to read / interpret the file (what the fields are, for the most part -- how they're encoded, what their meaning is, etc. Also, metadata about the document as a whole)

We've got a number of scientists that we'll likely never get them moved away from some form of ASCII tables, be they fixed-width, tab-delim or CSV.

Are there any community-wide standards for making text files self-documenting?

update : I'm not asking about XML. I want to know about community standards for making plain text files self-documenting. The scientists have absolutely no desire to package their data in some horribly bloated format (not even VOTable when fixed-width or tab delim ASCII tables do 99% of what they need.)

update 2 : as 'fixed-width ASCII tables' might not be used by all communities, they've been used for data transfer for decades, and they generally try to cram a lot into ~80 columns. For instance, the NOAA Solar Proton Events or CDAW Type II Radio Bursts. Then you have stuff that's not fixed-width, such as SPASE Event Lists (tab delim) or METAR (which is much more complex, as it's tokenized and some parts are optional so you have to use formating cues).

Answer 1

YAML frontmatter tends to be the generic standard for documenting text files. Parseable/compatible with JSON, easily human readable, easy to type in. It's used in content management systems, in combination with Markdown, to append metadata to blog posts in text files without the need for fully blown content management systems.

http://www.yaml.org/

Answer 2

There's a proposal called 'Linked CSV' which allows you to subtly insert bits to a CSV file so that it is simply transformed into RDF.

Documentation here - see in particular '2.2.4 Self Description'

The benefit over YAML is that it is still proper CSV, so will be read & written easily by the same tools as normal.

The benefit over the likes of Data Packages is that the metadata is in the same file, so less likely to get lost.

Linked CSV is focused on describing each column's type (e.g. country name or latitude or something). But you can also add metadata on lines at the top, putting 'meta' in the first column.

Here's the example of adding in a link to the index and the license for the data (although you can use anything in Dublic Core etc):

#,   country,                                 year,population
type,url,                                     time,integer
meta,index,                                   url, /populations
meta,license,                                 url, http://creativecommons.org/publicdomain/mark/1.0/
,    http://en.wikipedia.org/wiki/Afghanistan,1960,9616353
,    http://en.wikipedia.org/wiki/Afghanistan,1961,9799379
,    http://en.wikipedia.org/wiki/Afghanistan,1962,9989846
,    http://en.wikipedia.org/wiki/Afghanistan,1963,10188299

It is brand new so not established, although it is interesting Linked Data folk. There are no tools for it yet (AFAIK), but you could adapt this little syntactical idea for your purposes.

Answer 3

Have you looked at Apache AVRO (http://avro.apache.org/)? Avro relies on schemas. When Avro data is read, the schema used when writing it is always present. This permits each datum to be written with no per-value overheads, making serialization both fast and small. This also facilitates use with dynamic, scripting languages, since data, together with its schema, is fully self-describing.

When Avro data is stored in a file, its schema is stored with it, so that files may be processed later by any program. If the program reading the data expects a different schema this can be easily resolved, since both schemas are present

Answer 4

As obvious it is complex to have both data and metadata in the same file while keeping the structure simple. For sure XML can help a lot, but as far as I know there is no unique standard for the metainformation. What could help you in some ways could be RDF (in one of its many formats)... It's a long shot and it depends on what your scientists have to do with the data. But it could be the real change: having metadata about the dataset itself, as well as about each column can be really helpful. It doesn't come for free, but if you have some automatic operations in your processes, RDF can help you further automatizing operations.

Answer 5

going out on a limb here, but GitHub recently published a guide for citing Open Science data that sounds very similar to what you are seeking, including the desired audience, except i don't see anything about plain text, and since i'm only helping out here, i did not try the demo.
Digital Object Identifiers (DOI)s are used here in place of the plain text files...
https://guides.github.com/activities/citable-code/