11

One of the advantages of most scientific file formats is that you have a block of attached metadata that tells a reader how to read / interpret the file (what the fields are, for the most part -- how they're encoded, what their meaning is, etc. Also, metadata about the document as a whole)

We've got a number of scientists that we'll likely never get them moved away from some form of ASCII tables, be they fixed-width, tab-delim or CSV.

Are there any community-wide standards for making text files self-documenting?

update : I'm not asking about XML. I want to know about community standards for making plain text files self-documenting. The scientists have absolutely no desire to package their data in some horribly bloated format (not even VOTable when fixed-width or tab delim ASCII tables do 99% of what they need.)

update 2 : as 'fixed-width ASCII tables' might not be used by all communities, they've been used for data transfer for decades, and they generally try to cram a lot into ~80 columns. For instance, the NOAA Solar Proton Events or CDAW Type II Radio Bursts. Then you have stuff that's not fixed-width, such as SPASE Event Lists (tab delim) or METAR (which is much more complex, as it's tokenized and some parts are optional so you have to use formating cues).

  • 1
    related : opendata.stackexchange.com/q/157/263 – Joe May 12 '13 at 6:32
  • also, for an example of the sort of thing I'm looking for ... this came up about a year ago for the 'SPASE Event List Format' ... (Example, which I'm noticing is actually UTF, with non-ASCII characters in it ... doh) – Joe May 12 '13 at 6:35
  • Do you absolutely require the documentation / metadata to go into the data file or would you allow a simple (e.g. ascii text, maybe json) metadata file to describe the data file? – Rufus Pollock Jun 14 '13 at 10:12
  • @RufusPollock : I'd prefer some minimal information in the file (eg, an identifier to distinguish it from the rest of the collection), but the majority of it could be in a separate file, provided there were a way to give a link to it from within the text file. (or, I guess the standard could be 'you will have a README.txt file in the same directory'). – Joe Jun 14 '13 at 12:13
  • 1
    In that case the answer using Simple Data Format seems a good option - see also opendata.stackexchange.com/questions/157/… is pretty relevan – Rufus Pollock Jun 21 '13 at 15:05
3

There's a proposal called 'Linked CSV' which allows you to subtly insert bits to a CSV file so that it is simply transformed into RDF.

Documentation here - see in particular '2.2.4 Self Description'

The benefit over YAML is that it is still proper CSV, so will be read & written easily by the same tools as normal.

The benefit over the likes of Data Packages is that the metadata is in the same file, so less likely to get lost.

Linked CSV is focused on describing each column's type (e.g. country name or latitude or something). But you can also add metadata on lines at the top, putting 'meta' in the first column.

Here's the example of adding in a link to the index and the license for the data (although you can use anything in Dublic Core etc):

#,   country,                                 year,population
type,url,                                     time,integer
meta,index,                                   url, /populations
meta,license,                                 url, http://creativecommons.org/publicdomain/mark/1.0/
,    http://en.wikipedia.org/wiki/Afghanistan,1960,9616353
,    http://en.wikipedia.org/wiki/Afghanistan,1961,9799379
,    http://en.wikipedia.org/wiki/Afghanistan,1962,9989846
,    http://en.wikipedia.org/wiki/Afghanistan,1963,10188299

It is brand new so not established, although it is interesting Linked Data folk. There are no tools for it yet (AFAIK), but you could adapt this little syntactical idea for your purposes.

2

Have you looked at Apache AVRO (http://avro.apache.org/)? Avro relies on schemas. When Avro data is read, the schema used when writing it is always present. This permits each datum to be written with no per-value overheads, making serialization both fast and small. This also facilitates use with dynamic, scripting languages, since data, together with its schema, is fully self-describing.

When Avro data is stored in a file, its schema is stored with it, so that files may be processed later by any program. If the program reading the data expects a different schema this can be easily resolved, since both schemas are present

  • Interesting ... it has a ton of great features, but it seems to be JSON-based for the data payload, not just the header. (although, their metadata schema looks quite rich ... and they even documentation for enums and such) – Joe May 12 '13 at 15:00
2

YAML frontmatter tends to be the generic standard for documenting text files. Parseable/compatible with JSON, easily human readable, easy to type in. It's used in content management systems, in combination with Markdown, to append metadata to blog posts in text files without the need for fully blown content management systems.

http://www.yaml.org/

1

As obvious it is complex to have both data and metadata in the same file while keeping the structure simple. For sure XML can help a lot, but as far as I know there is no unique standard for the metainformation. What could help you in some ways could be RDF (in one of its many formats)... It's a long shot and it depends on what your scientists have to do with the data. But it could be the real change: having metadata about the dataset itself, as well as about each column can be really helpful. It doesn't come for free, but if you have some automatic operations in your processes, RDF can help you further automatizing operations.

  • Yes, something like the NOAA SRS files are small enough to be re-cast as XML ... but I've got million-record bright point catalogs ... the scientists in our community would prefer to use FITS tables or IDL save files before using XML. – Joe May 12 '13 at 11:22
1

going out on a limb here, but GitHub recently published a guide for citing Open Science data that sounds very similar to what you are seeking, including the desired audience, except i don't see anything about plain text, and since i'm only helping out here, i did not try the demo.
Digital Object Identifiers (DOI)s are used here in place of the plain text files...
https://guides.github.com/activities/citable-code/

  • Thanks, but that's not quite what I'm looking for ... they take the files and package them (not sure if they're using BagIt or something else), attaching the metadata in the process. I'm actually trying to find formats where the metadata is in the data file, not just in a tarball / zip archive with it. And if you're interested in software citation, also see software.ac.uk – Joe Aug 27 '14 at 15:45
  • microformats them up! combine them with data- attributes to create the taxonomies you desire – albert Aug 28 '14 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.