12

One of the advantages of most scientific file formats is that you have a block of attached metadata that tells a reader how to read / interpret the file (what the fields are, for the most part -- how they're encoded, what their meaning is, etc. Also, metadata about the document as a whole)

We've got a number of scientists that we'll likely never get them moved away from some form of ASCII tables, be they fixed-width, tab-delim or CSV.

Are there any community-wide standards for making text files self-documenting?

update : I'm not asking about XML. I want to know about community standards for making plain text files self-documenting. The scientists have absolutely no desire to package their data in some horribly bloated format (not even VOTable when fixed-width or tab delim ASCII tables do 99% of what they need.)

update 2 : as 'fixed-width ASCII tables' might not be used by all communities, they've been used for data transfer for decades, and they generally try to cram a lot into ~80 columns. For instance, the NOAA Solar Proton Events or CDAW Type II Radio Bursts. Then you have stuff that's not fixed-width, such as SPASE Event Lists (tab delim) or METAR (which is much more complex, as it's tokenized and some parts are optional so you have to use formating cues).

5
  • 1
    related : opendata.stackexchange.com/q/157/263
    – Joe
    May 12, 2013 at 6:32
  • also, for an example of the sort of thing I'm looking for ... this came up about a year ago for the 'SPASE Event List Format' ... (Example, which I'm noticing is actually UTF, with non-ASCII characters in it ... doh)
    – Joe
    May 12, 2013 at 6:35
  • Do you absolutely require the documentation / metadata to go into the data file or would you allow a simple (e.g. ascii text, maybe json) metadata file to describe the data file? Jun 14, 2013 at 10:12
  • @RufusPollock : I'd prefer some minimal information in the file (eg, an identifier to distinguish it from the rest of the collection), but the majority of it could be in a separate file, provided there were a way to give a link to it from within the text file. (or, I guess the standard could be 'you will have a README.txt file in the same directory').
    – Joe
    Jun 14, 2013 at 12:13
  • 1
    In that case the answer using Simple Data Format seems a good option - see also opendata.stackexchange.com/questions/157/… is pretty relevan Jun 21, 2013 at 15:05

5 Answers 5

3

YAML frontmatter tends to be the generic standard for documenting text files. Parseable/compatible with JSON, easily human readable, easy to type in. It's used in content management systems, in combination with Markdown, to append metadata to blog posts in text files without the need for fully blown content management systems.

http://www.yaml.org/

3

There's a proposal called 'Linked CSV' which allows you to subtly insert bits to a CSV file so that it is simply transformed into RDF.

Documentation here - see in particular '2.2.4 Self Description'

The benefit over YAML is that it is still proper CSV, so will be read & written easily by the same tools as normal.

The benefit over the likes of Data Packages is that the metadata is in the same file, so less likely to get lost.

Linked CSV is focused on describing each column's type (e.g. country name or latitude or something). But you can also add metadata on lines at the top, putting 'meta' in the first column.

Here's the example of adding in a link to the index and the license for the data (although you can use anything in Dublic Core etc):

#,   country,                                 year,population
type,url,                                     time,integer
meta,index,                                   url, /populations
meta,license,                                 url, http://creativecommons.org/publicdomain/mark/1.0/
,    http://en.wikipedia.org/wiki/Afghanistan,1960,9616353
,    http://en.wikipedia.org/wiki/Afghanistan,1961,9799379
,    http://en.wikipedia.org/wiki/Afghanistan,1962,9989846
,    http://en.wikipedia.org/wiki/Afghanistan,1963,10188299

It is brand new so not established, although it is interesting Linked Data folk. There are no tools for it yet (AFAIK), but you could adapt this little syntactical idea for your purposes.

2

Have you looked at Apache AVRO (http://avro.apache.org/)? Avro relies on schemas. When Avro data is read, the schema used when writing it is always present. This permits each datum to be written with no per-value overheads, making serialization both fast and small. This also facilitates use with dynamic, scripting languages, since data, together with its schema, is fully self-describing.

When Avro data is stored in a file, its schema is stored with it, so that files may be processed later by any program. If the program reading the data expects a different schema this can be easily resolved, since both schemas are present

1
  • Interesting ... it has a ton of great features, but it seems to be JSON-based for the data payload, not just the header. (although, their metadata schema looks quite rich ... and they even documentation for enums and such)
    – Joe
    May 12, 2013 at 15:00
1

As obvious it is complex to have both data and metadata in the same file while keeping the structure simple. For sure XML can help a lot, but as far as I know there is no unique standard for the metainformation. What could help you in some ways could be RDF (in one of its many formats)... It's a long shot and it depends on what your scientists have to do with the data. But it could be the real change: having metadata about the dataset itself, as well as about each column can be really helpful. It doesn't come for free, but if you have some automatic operations in your processes, RDF can help you further automatizing operations.

1
  • Yes, something like the NOAA SRS files are small enough to be re-cast as XML ... but I've got million-record bright point catalogs ... the scientists in our community would prefer to use FITS tables or IDL save files before using XML.
    – Joe
    May 12, 2013 at 11:22
1

going out on a limb here, but GitHub recently published a guide for citing Open Science data that sounds very similar to what you are seeking, including the desired audience, except i don't see anything about plain text, and since i'm only helping out here, i did not try the demo.
Digital Object Identifiers (DOI)s are used here in place of the plain text files...
https://guides.github.com/activities/citable-code/

2
  • Thanks, but that's not quite what I'm looking for ... they take the files and package them (not sure if they're using BagIt or something else), attaching the metadata in the process. I'm actually trying to find formats where the metadata is in the data file, not just in a tarball / zip archive with it. And if you're interested in software citation, also see software.ac.uk
    – Joe
    Aug 27, 2014 at 15:45
  • microformats them up! combine them with data- attributes to create the taxonomies you desire
    – albert
    Aug 28, 2014 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.