Data management practices

Question

Software has code repositories and package managers, so installation and maintenance are done in one line (or click).

What are practices for managing data, instead? Apart from manually organized directories with readme files inside, I found a couple solutions:

• drake from Factual, https://github.com/Factual/drake
• CKAN from OKFN, https://okfn.org/projects/ckan/
• Zotero and other librarian tools, so you save datasets instead of books with them

However, drake is mostly about workflows, not maintenance. And CKAN is server-based solution for which I couldn't find a local "package manager".

What would such a package manager ideally do:

• downloading datasets from source in one click
• supporting multiple formats and conversion
• maintaining the directory structure
• updating the data from the repositories
• supporting aliases for datasets
• holding meta information about the datasets
• tags

Have you seen anything like this? If not, maybe you'd share your practices of maintaining the data manually?

Answer 1

• R, the open source and cross-platform statistics software has something similar where sometimes packages/plugins for R include sample or test data:

  • See this SO question: https://stackoverflow.com/questions/12391195/include-data-examples-in-developing-r-packages
  • and this info page: http://r-pkgs.had.co.nz/data.html

• OKFN (the same folks behind CKAN) has something they called Data Packages (http://dataprotocols.org/data-packages/) which sounds like most of what you're looking for.

  • Ecosystem image from Data Packages website embedded below: