13

I work with a a couple of small non-profit genealogical and historical groups and we are interested in releasing some of the datasets we've compiled over the years as open data. This information is already freely searchable through our online databases, which use Apache Solr and/or MySQL on the backend, but we'd like to start explicitly recognizing that this data is in the public domain and releasing the sets for individual download as CSV's.

Obviously, it's not very likely that the old Austro-Hungarian Empire is going to resurrect itself just so that it can set up a Socrata or CKAN portal for its old town-by-town tax rolls and birth indices. :-) So, it looks like I may be the one creating a new website later this year or early next year to collect and distribute genealogical/historical datasets such as these. Ideally, such a site would have detailed metadata available for the datasets, including how the data was transcribed (by hand from digital photographs or scans, usually), where the original copies are stored (libraries, archives), and would correctly label the data as public domain, since it was usually created by government entities, most of which do not exist anymore. Some kind of versioning history would be nice too, in case any mistakes or typos are later discovered and fixed.

I think I've been in touch with all the historical/genealogical groups that I know of that would be interested in contributing to such a new endeavor, but I want to make sure I am aware of all the open data groups that might have an interest too. What open data groups (besides Code4Lib, ODI, etc.) would be good for me to contact? What are some things I would need to keep in mind if I created such an online archive? What problems do you foresee with this idea?

Finally, suggestions for a name for this new site would be most appreciated. ("Open Data of the Dead" is probably way too creepy, right?)

  • Did you ever publish the datasets on a website? If so, what format and licensing did you choose? – Andrew - OpenGeoCode May 19 '14 at 16:58
  • As of May 2014, not yet, sorry. But that's because we've been focusing our time on formatting and publishing the data in freely searchable form in our search engine. 280,000+ records and counting... – Asparagirl May 29 '14 at 1:16
5

Publishing the Data

If you don't want to go down the full CKAN-style route a really simple option if you've got a bunch of CSV files is just to put them online and turn them into "simple data format" data packages by adding a tiny bit of metadata in the form of a datapackage.json. Building a catalog out of this is really easy (and can be done with just a bit of JS+HTML!).

Simple Data Format

For details you can see this simple introduction to simple data format for more information. Here's a simple example:

Here's an example of a minimal simple data format dataset:

There are 2 just files, the data file data.csv and the datapackage.json:

data.csv
datapackage.json

data.csv looks like:

var1,var2,var3
A,1,2.5
B,3,4.3

That is there are 3 fields (columns) and 2 rows of data.

A simple datapackage.json for this data would be:

{
  "name": "my-dataset",
  # here we list the data files in this dataset
  "resources": [
    {
      "path": "data.csv",
      "schema": {
        "fields": [
          {
            "id": "var1",
            "type": "string"
          },
          {
            "id": "var2",
            "type": "integer"
          },
          {
            "id": "var3",
            "type": "number"
          }
        ]
      }
    }
  ]
}

People to contact

Suggest these type of folks (you may already know them):

(Disclosure: I helped write the simple data format spec and am a member of the Open Knowledge Foundation!)

3

This is obviously a very big topic and any answer will be incomplete. I would suggest a few specific issues to consider.

  1. Licensing of the collection as a whole for fruits-of-labor jurisdictions. Now, I prefer the BSD-style licenses (no sublicensing, but derivatives can be licensed under any other terms). In general, you have to figure that if someone takes your data and markets a larger commercial database in the UK, it gives you some pointers for what data to focus on collecting.

  2. Domain Names. How about something like Open Ancestors or Open Regional History?

  3. Web service access? Do you want to solicit donations by allowing live integration via web services for a fee (vs possibly a free data dump and online search interface)? If so what sort of organization would be involved in managing the money? Adding a value added service and putting the money towards continued work is not a bad way to fund it.

  • Oooh, I like "Open Ancestors". Or maybe "Open Past" to be broader? We would want to be able to explicitly include datasets that are not just genealogical, but also sociological or historical. (Real example: a transcribed list of everyone who was a registered voter in a formerly-Polish now-Ukrainian town in the 1920's.) – Asparagirl May 15 '13 at 16:56
2

Just to be sure you are aware, the U.S. government releases genealogy data at a 72-year time lag (to help protect individual data of living persons). For example, information about the 1940 census for genealogy can be found at http://1940census.archives.gov/ and overall data at the Census is at http://www.census.gov/main/www/access.html

  • 3
    Sorry, but this is not exactly true, and not true in all cases. First of all, it's a 72-year privacy window, not 70; the 1940 US Federal Census went online in 2012. The rule only applies to the decennial census, not to any off-year state or local census records that may exist, and not to any other genealogical data. The rule has also been violated by the US government at least once before, when Japanese-Americans were identified using their supposedly private responses to the 1940 US Federal Census so that they could be rounded up for internment during WWII. – Asparagirl May 15 '13 at 16:51
  • Thank you for the clarification about the time lag. I've edited my post to be accurate. – Jeanne Holm May 15 '13 at 18:33
-1

The National Technical Information Service (NTIS.gov) is the official distributor of the Social Security Administration's (SSA) Death Master File (DMF) and contains over 85 million records of deaths, reported to the SSA, from 1936 to present, updated on a weekly basis as information becomes available. Name, SSN, date of birth, date of death, state of residence are included. It is available to the public for purchase.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.