25

We publish most of our machine-readable open data in CSV format.

What are best practices and/or standards to publish data dictionaries (e.g. definitions of columns in CSV files including human-readable names, data types, possible values and their definitions)?

Currently we use subset of FGDC Metadata Standard since it has elements to define metadata for each column.

Are there simpler (or even plain-text) formats for this?

12 Answers 12

21

I'd suggest using JSON Table Schema: http://www.dataprotocols.org/en/latest/json-table-schema.html

It's:

  • JSON-based
  • Super simple
  • Extendable

Here's a rough outline:

  # fields is an ordered list of field descriptors
  # one for each field (column) in the data
  "fields": [
    # a field-descriptor
    {
      "id": "field unique name / id",
      "label": "A nicer human readable label for the field",
      "type": "A string specifying the type",
      "format": "A string specifying a format",
      "description": "A description for the field"
      ...
    },
    ... more field descriptors
  ]
  • Points for simplicity. Suggestions on dealing with field value definitions? Possibilities for linking to definitions / resources on the web? e.g. a standard for a type, for a format string - or is this just meant to be human readable? – user468648 May 13 '13 at 13:41
  • Thanks Rufus, exactly what I was looking for! Any links to samples of usage of JSON Table Schema in .gov open data catalogs? – Dmitry Kachaev May 13 '13 at 14:18
  • @user468648 In terms of extending JSON Table Schema, it may make sense for it to become a dialect of JSON Schema json-schema.org, that way it can inherit a lot of the good work that has gone into that. – James McKinney May 22 '13 at 1:22
  • This looks like a cool spec, but I don''t see many tools that work with it two years later. Where is this getting traction? – chicks Sep 21 '15 at 21:50
  • 1
    @chicks Have you looked at data.okfn.org/tools - there is now support and implementations in several languages (Ruby, Python, R, Java etc) and platforms (MatLab, CKAN etc) – Rufus Pollock Nov 8 '15 at 12:50
11

While definitely not yet an established practice, another proposal to consider might be Jeni Tennison's Linked CSV (see examples), which adds "prolog lines" to describe the schema used in CSV files.

7

While SKOS certainly might be the best way to represent this information it does require more effort than I believe most will be willing to provide. Couldn't we start with a simple, practical, form for providing a data dictionary with another CSV file with 4 mandatory columns. DatasetName, FieldName, FieldValue or Code, ValueDefinition and an optional hyperlink. I think this is more common practice (well excepting the hyperlink).

e.g. DatasetName, FieldName, FieldValue, ValueDefinition, RelevantHyperlink WHOTBburdenestimates, NULL, NULL, NULL, http://www.who.int/tb/country/data/download/en/ WHOTBburdenestimates, country, NULL, Country identification, http://www.iso.org/iso/home/standards/country_codes/iso-3166-1_decoding_table.htm WHOTBburdenestimates, country, AF, Afghanistan,NULL WHOTBburdenestimates, country, AL, Albania,NULL ... WHOTBburdenestimates, source_mdr_new, DRS, drug resistance surveillance or survey,NULL WHOTBburdenestimates, source_mdr_new, MOD, statistical model,NULL

  • 3
    SKOS is the right answer for this problem -- if you think it's too much effort, maybe we need to focus on better tools for authoring SKOS (at least for the simples cases) rather than some less capable competing format. – Joe May 12 '13 at 6:39
5

I wouldn't dismiss XML so lightly. In the first place, given the natural vagaries of data transmission (especially when considered world-wide), information should be put in XML to simplify error-catching -- the start- and end-point of every datum is unambiguously identified. In the second place, if your XML is self-documenting, as the XML spec intends, it should be a straightforward matter to transform your data to an industry standard that appears at sometime in the future.

However, metadata may better be represented in a relational technology (such as OWL or Topic Maps) that does not require rigid and pre-accepted definitions of what your data is and how its pieces relate.

So you might want to look at something like SKOS for your metadata.

  • I'm talking specifically about data in CSV format (comma separated values). So I'm looking for simplicity in metadata format. If my data was in XML then XML Schema, et al will be a no brainer. – Dmitry Kachaev May 10 '13 at 3:15
5

The National Information Exchange Model (NIEM) is an XML-based system for defining "data in motion" (i.e., an on-the-wire format.) What distinguishes NIEM is 2 things: it is for standardizing the semantics of the exchange, not just the syntax; and it as much a process model as it is a technical model. That is, it (the NIEM organization) has developed a process for participants in a community-of-interest to define a set of meaningful business data exchanges, structured as XML documents and called IEPD's (I think that's "information exchange package definitions"), which participants can then implement in Web services, etc.

NIEM started out as a data-exchange standardization effort within the US State and local public safety community (i.e., law-enforcement and related services) , but is now widely used in the US Federal Government as a recommended data model standard. For example, HHS is the NIEM domain manager for health-related exchanges. And DOD recently decided to convert their metadata efforts to NIEM. NIEM is not a public standard (still technically a US Government interagency project) for reasons I don't fully understand, but I would expect it to transition in that direction at some point.

One step in that direction is a recent effort within OMG to "translate" NEIM IEPD exchanges into UML, which makes a lot of sense since the heart of the NIEM process is defining data exchanges that support business transactions, which are typically created to support standard business processes.

For more info, see https://www.niem.gov/Pages/default.aspx

4

I have recently started using the Linked CSV proposed standard for generating CSV files from plural data sources. Below is a vocabulary definition for the columns/data types I am using. Perhaps others will find this useful/interesting approach:

http://www.opengeocode.org/cude1.1/LinkedCSV-Vocab.php

Update: the above link throws a 404, however it is viewable via the Wayback Machine:
http://web.archive.org/web/20140625055123/http://opengeocode.org/cude1.1/LinkedCSV-Vocab.php

Version 1.2 is live (and I'm assuming replaces 1.1):
http://www.opengeocode.org/cude1.2/LinkedCSV-Vocab.php

3

Chicago has been doing a pretty good job describing their data. The about page for the food description has is pretty thorough, if you count links to external resources.

https://data.cityofchicago.org/Health-Human-Services/Food-Inspections/4ijn-s7e5/about

Good points:

  • It describes how the data is entered
  • It links to a detailed description of the meaning of every field
  • It discusses the quality of the data
  • 1
    I have to admit, I'm much less impressed with Chicago's data documentation. You've identified a one good example, but it's far from typical. When metadata is present, it's offered in a myriad of forms, and none of it is structured. – Joe Germuska May 15 '13 at 13:59
3

The Data Documentation Initiative provides the DDI metadata standard for describing data dictionaries.

The DDI standard is broadly used and maintained by a large consortium of Universities and government organizations.

2

This is a problem I've had to deal with too. Check out the OMG Standard which tries to define some basic field types for location-based open data.

www.OMGStandard.com

Does a good job of keeping things generic enough to apply to lots of different data, but specific enough to be filterable and comparable across data sets.

2

It looks like Jeni Tennison took her Linked CSV spec to the W3C, and with some help from Gregg Kellogg and Ivan Herman it evolved into the Model for Tabular Data and Metadata on the Web and the Metadata Vocabulary for Tabular Data that both appeared as a W3C candidate recommendation on 16 July, 2015. Altogether, four CSV-related recommendations have appeared; see the announcement.

PS: If you do not want to create Linked Data with all those URLs in it, then take a look at the Tabular Data Package standard of the Open Knowledge Foundation. It adopts JSON Table Schema and adds some useful conventions.

1

The National Information Exchange Model (NIEM), which is XML-based, has a variety of schemas for various industries. If one does not yet exist for your industry or needs, you can give input into the development of a suitable model.

  • I'm looking for generic metadata format. My data is already in CSV format and is not standardized beyond being anything tabular (e.g. NIEM data will be in XML and each industry have to standardize on schema before-hand) – Dmitry Kachaev May 9 '13 at 18:13
1

There is a data catalog entity define in schema.org, with defined attributes to describe the data-set.

Good Points:

  • schema already defined
  • supported by all majors search engine
  • flexible schema that can be extended if needed.
  • That might be useful for a data catalog (a list of datasets) but not a data dictionary (a description of a dataset) – Joe May 17 '13 at 0:05
  • Right, there is also a data set entity define in the model. – magdmartin May 17 '13 at 6:09
  • you would need a 'data set column' or similar defined to be able to build a data dictionary – Joe May 17 '13 at 6:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.