We publish most of our machine-readable open data in CSV format.
What are best practices and/or standards to publish data dictionaries (e.g. definitions of columns in CSV files including human-readable names, data types, possible values and their definitions)?
Currently we use subset of FGDC Metadata Standard since it has elements to define metadata for each column.
Are there simpler (or even plain-text) formats for this?
I'd suggest using JSON Table Schema: https://specs.frictionlessdata.io/table-schema
It's:
Here's a rough outline:
# fields is an ordered list of field descriptors
# one for each field (column) in the data
"fields": [
# a field-descriptor
{
"id": "field unique name / id",
"label": "A nicer human readable label for the field",
"type": "A string specifying the type",
"format": "A string specifying a format",
"description": "A description for the field"
...
},
... more field descriptors
]
While definitely not yet an established practice, another proposal to consider might be Jeni Tennison's Linked CSV (see examples), which adds "prolog lines" to describe the schema used in CSV files.
While SKOS certainly might be the best way to represent this information it does require more effort than I believe most will be willing to provide. Couldn't we start with a simple, practical, form for providing a data dictionary with another CSV file with 4 mandatory columns. DatasetName, FieldName, FieldValue or Code, ValueDefinition and an optional hyperlink. I think this is more common practice (well excepting the hyperlink).
e.g.
DatasetName, FieldName, FieldValue, ValueDefinition, RelevantHyperlink
WHOTBburdenestimates, NULL, NULL, NULL, http://www.who.int/tb/country/data/download/en/
WHOTBburdenestimates, country, NULL, Country identification, http://www.iso.org/iso/home/standards/country_codes/iso-3166-1_decoding_table.htm
WHOTBburdenestimates, country, AF, Afghanistan,NULL
WHOTBburdenestimates, country, AL, Albania,NULL
...
WHOTBburdenestimates, source_mdr_new, DRS, drug resistance surveillance or survey,NULL
WHOTBburdenestimates, source_mdr_new, MOD, statistical model,NULL
I wouldn't dismiss XML so lightly. In the first place, given the natural vagaries of data transmission (especially when considered world-wide), information should be put in XML to simplify error-catching -- the start- and end-point of every datum is unambiguously identified. In the second place, if your XML is self-documenting, as the XML spec intends, it should be a straightforward matter to transform your data to an industry standard that appears at sometime in the future.
However, metadata may better be represented in a relational technology (such as OWL or Topic Maps) that does not require rigid and pre-accepted definitions of what your data is and how its pieces relate.
So you might want to look at something like SKOS for your metadata.
The National Information Exchange Model (NIEM) is an XML-based system for defining "data in motion" (i.e., an on-the-wire format.) What distinguishes NIEM is 2 things: it is for standardizing the semantics of the exchange, not just the syntax; and it as much a process model as it is a technical model. That is, it (the NIEM organization) has developed a process for participants in a community-of-interest to define a set of meaningful business data exchanges, structured as XML documents and called IEPD's (I think that's "information exchange package definitions"), which participants can then implement in Web services, etc.
NIEM started out as a data-exchange standardization effort within the US State and local public safety community (i.e., law-enforcement and related services) , but is now widely used in the US Federal Government as a recommended data model standard. For example, HHS is the NIEM domain manager for health-related exchanges. And DOD recently decided to convert their metadata efforts to NIEM. NIEM is not a public standard (still technically a US Government interagency project) for reasons I don't fully understand, but I would expect it to transition in that direction at some point.
One step in that direction is a recent effort within OMG to "translate" NEIM IEPD exchanges into UML, which makes a lot of sense since the heart of the NIEM process is defining data exchanges that support business transactions, which are typically created to support standard business processes.
For more info, see https://www.niem.gov/Pages/default.aspx
I have recently started using the Linked CSV proposed standard for generating CSV files from plural data sources. Below is a vocabulary definition for the columns/data types I am using. Perhaps others will find this useful/interesting approach:
http://www.opengeocode.org/cude1.1/LinkedCSV-Vocab.php
Update: the above link throws a 404, however it is viewable via the Wayback Machine:
http://web.archive.org/web/20140625055123/http://opengeocode.org/cude1.1/LinkedCSV-Vocab.php
Version 1.2 is live (and I'm assuming replaces 1.1):
http://www.opengeocode.org/cude1.2/LinkedCSV-Vocab.php
Chicago has been doing a pretty good job describing their data. The about page for the food description has is pretty thorough, if you count links to external resources.
https://data.cityofchicago.org/Health-Human-Services/Food-Inspections/4ijn-s7e5/about
Good points:
The Data Documentation Initiative provides the DDI metadata standard for describing data dictionaries.
The DDI standard is broadly used and maintained by a large consortium of Universities and government organizations.
It looks like Jeni Tennison took her Linked CSV spec to the W3C, and with some help from Gregg Kellogg and Ivan Herman it evolved into the Model for Tabular Data and Metadata on the Web and the Metadata Vocabulary for Tabular Data that both appeared as a W3C candidate recommendation on 16 July, 2015. Altogether, four CSV-related recommendations have appeared; see the announcement.
PS: If you do not want to create Linked Data with all those URLs in it, then take a look at the Tabular Data Package standard of the Open Knowledge Foundation. It adopts JSON Table Schema and adds some useful conventions.
This is a problem I've had to deal with too. Check out the OMG Standard which tries to define some basic field types for location-based open data.
Does a good job of keeping things generic enough to apply to lots of different data, but specific enough to be filterable and comparable across data sets.
The National Information Exchange Model (NIEM), which is XML-based, has a variety of schemas for various industries. If one does not yet exist for your industry or needs, you can give input into the development of a suitable model.
There is a data catalog entity define in schema.org, with defined attributes to describe the data-set.
Good Points:
