Releasing old historical/genealogical datasets as open data

Question

I work with a a couple of small non-profit genealogical and historical groups and we are interested in releasing some of the datasets we've compiled over the years as open data. This information is already freely searchable through our online databases, which use Apache Solr and/or MySQL on the backend, but we'd like to start explicitly recognizing that this data is in the public domain and releasing the sets for individual download as CSV's.

Obviously, it's not very likely that the old Austro-Hungarian Empire is going to resurrect itself just so that it can set up a Socrata or CKAN portal for its old town-by-town tax rolls and birth indices. :-) So, it looks like I may be the one creating a new website later this year or early next year to collect and distribute genealogical/historical datasets such as these. Ideally, such a site would have detailed metadata available for the datasets, including how the data was transcribed (by hand from digital photographs or scans, usually), where the original copies are stored (libraries, archives), and would correctly label the data as public domain, since it was usually created by government entities, most of which do not exist anymore. Some kind of versioning history would be nice too, in case any mistakes or typos are later discovered and fixed.

I think I've been in touch with all the historical/genealogical groups that I know of that would be interested in contributing to such a new endeavor, but I want to make sure I am aware of all the open data groups that might have an interest too. What open data groups (besides Code4Lib, ODI, etc.) would be good for me to contact? What are some things I would need to keep in mind if I created such an online archive? What problems do you foresee with this idea?

Finally, suggestions for a name for this new site would be most appreciated. ("Open Data of the Dead" is probably way too creepy, right?)

Answer 1

Publishing the Data

If you don't want to go down the full CKAN-style route a really simple option if you've got a bunch of CSV files is just to put them online and turn them into "simple data format" data packages by adding a tiny bit of metadata in the form of a datapackage.json. Building a catalog out of this is really easy (and can be done with just a bit of JS+HTML!).

Simple Data Format

For details you can see this simple introduction to simple data format for more information. Here's a simple example:

Here's an example of a minimal simple data format dataset:

There are 2 just files, the data file data.csv and the datapackage.json:

data.csv
datapackage.json

data.csv looks like:

var1,var2,var3
A,1,2.5
B,3,4.3

That is there are 3 fields (columns) and 2 rows of data.

A simple datapackage.json for this data would be:

{
  "name": "my-dataset",
  # here we list the data files in this dataset
  "resources": [
    {
      "path": "data.csv",
      "schema": {
        "fields": [
          {
            "id": "var1",
            "type": "string"
          },
          {
            "id": "var2",
            "type": "integer"
          },
          {
            "id": "var3",
            "type": "number"
          }
        ]
      }
    }
  ]
}

People to contact

Suggest these type of folks (you may already know them):

• Folks at the Open Knowledge Foundation - e.g. worth pinging the okfn-discuss mailing list (there have been several discussions about genealogical data here over the years)
• http://www.opengenalliance.org/

(Disclosure: I helped write the simple data format spec and am a member of the Open Knowledge Foundation!)

Answer 2

This is obviously a very big topic and any answer will be incomplete. I would suggest a few specific issues to consider.

1. Licensing of the collection as a whole for fruits-of-labor jurisdictions. Now, I prefer the BSD-style licenses (no sublicensing, but derivatives can be licensed under any other terms). In general, you have to figure that if someone takes your data and markets a larger commercial database in the UK, it gives you some pointers for what data to focus on collecting.

2. Domain Names. How about something like Open Ancestors or Open Regional History?

3. Web service access? Do you want to solicit donations by allowing live integration via web services for a fee (vs possibly a free data dump and online search interface)? If so what sort of organization would be involved in managing the money? Adding a value added service and putting the money towards continued work is not a bad way to fund it.

Answer 3

Just to be sure you are aware, the U.S. government releases genealogy data at a 72-year time lag (to help protect individual data of living persons). For example, information about the 1940 census for genealogy can be found at http://1940census.archives.gov/ and overall data at the Census is at http://www.census.gov/main/www/access.html

Answer 4

The National Technical Information Service (NTIS.gov) is the official distributor of the Social Security Administration's (SSA) Death Master File (DMF) and contains over 85 million records of deaths, reported to the SSA, from 1936 to present, updated on a weekly basis as information becomes available. Name, SSN, date of birth, date of death, state of residence are included. It is available to the public for purchase.