I know you can put CSV or XML in Git already, but there are issues for those wishing to collaborate on a dataset for creation, cleaning it up, pull requests, etc. Are there alternative version control systems that suit data better?
The key improvement I'd want is to Diff/Merge more intelligently. e.g. in CSV rather than line vs line comparison, it would do cell vs cell.
And ordering is usually not significant, e.g. rows in a CSV, whereas Git does care and presents the user with 'conflicts'.
I recently stumbled on this article by the Open Knowledge Foundation regarding the design of a graphical interface to diff tabular data called daff.
It can also be tested and forked on GitHub.
Basic answer afaik is: "No".
As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures.
Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as suggested by Deer Hunter).
Fundamentally git or hg or a given VCS is built around a diff format and a merge protocol. Get these right and much of the rest follows. The basic 3 options you have are:
So far, whilst imperfect (1) beats out anything I've really seen for 2 or 3.
Update: I've written some of this up in much more detail in this post which recommends using simple text-based formats and then using version control like Git - http://blog.okfn.org/2013/07/02/git-and-github-for-data/
Update: there is also the Frictionless Data project which is working to making this kind of thing easier. In particular, the "Core Datasets" part of that is actively working to maintain a set of "core" datasets in version control on Github - see http://data.okfn.org/roadmap/core-datasets and https://github.com/datasets
Here's a long post from 3y ago (my how time flies) on why we would want this and what's difficult about it:
http://blog.okfn.org/2010/07/12/we-need-distributed-revisionversion-control-for-data/
Here's a summary of concepts around versioning for data: http://www.dataprotocols.org/en/latest/revisioning-data.html
The section entitled "diffing and merging" in this post has a summary of the basic options.
There is a project dat that addresses what you are looking for. It is now in beta, featuring a command line tool with similar design like git. There are also APIs for Javascript (dat-core), Python (datpy) and R (rdat).
Supported input data formats for tabular data are (as of August 2015): CSV, TSV, JSON, and newline delimited JSON. It also supports versioning binary blobs (e.g. image data) as a simple key-value storage.
There is a whitepaper that lists the design goals and workflows that are now working or expected to work with dat. Like git, it already supports distributed versioning with forking and merging.
For trying it out, there is an interactive tutorial at http://try-dat.com/.
Also take a look at "dat jawn: 'Git for Tabular Data'" at https://github.com/CodeForPhilly/jawn .
Please note that git has two configuration commands:
git config filter.<driver>.clean
git config filter.<driver>.smudge
The clean filter can be used e.g. to sort all records in a CSV file except the top row, thus making re-ordering irrelevant.
Quoting gitattributes(5) :
A filter driver consists of a clean command and a smudge command, either of which can be left unspecified. Upon checkout, when the smudge command is specified, the command is fed the blob object from its standard input, and its standard output is used to update the worktree file. Similarly, the clean command is used to convert the contents of worktree file upon checkin.
A missing filter driver definition in the config is not an error but makes the filter a no-op passthru.
The content filtering is done to massage the content into a shape that is more convenient for the platform, filesystem, and the user to use. The key phrase here is "more convenient" and not "turning something unusable into usable". In other words, the intent is that if someone unsets the filter driver definition, or does not have the appropriate filter program, the project should still be usable.
Can you provide a more comprehensive list of the pain points you're experiencing with Git? Git works great for data.
In addition to the config flag Deer Hunter mentioned for sorting, you can also teach git to diff on a word-by-word (rather than line-by-line basis), by simply passing the --word-diff flag.
git diff --word-diff, you can also use git diff --color-words, which is much prettier :)
Commented
Jun 11, 2013 at 21:51
Since this question was originally asked, the Dat project has made a lot of progress. Originally conceived by Max Ogden, it now has several other developers working on it.
dat is an open source tool that enables the sharing of large datasets, the goal being a collaboration flow similar to what git offers for source code. As a team we have a bias towards supporting scientific + research data use cases.
The screencast linked from the home page (or here on YouTube) is worth watching if you're interested in this kind of stuff.
Another open-source one that has popped up is DVC
https://github.com/iterative/dvc
Data Science Version Control or DVC is an open-source tool for data science and machine learning projects. With a simple and flexible Git-like architecture and interface
DVC is compatible with Git for storing code and the dependency graph (DAG), but not data files cache. To store and share data files cache DVC supports remotes - any cloud (S3, Azure, Google Cloud, etc) or any on-premise network storage (via SSH, for example).
Yes there is one, it is called noms
https://github.com/attic-labs/noms
Noms is a decentralized database philosophically descendant from the Git version control system.
Git's diff-ing and merge-ing functionality can be extended using custom drivers. I just wrote a custom merging driver for json files that merges based on location in the tree instead of on a line-by-line basis. It's only 96 lines of coffee-script (with comments): https://gist.github.com/jphaas/ad7823b3469aac112a52
I can imagine doing something similar for CSV files. The algorithm should be even simpler since CSV isn't a recursive format.
Given how easy it is to extend Git like this, I suggest going down the custom driver path rather than trying to massage the data into a line-by-line format.
This topic was also discussed sometime ago at answers.semanticweb.com, see
Here are also some links to some semantic versioning approaches:
I don't know if you are still looking for a git for data but there is Core Object that present themselves as: "a version-controlled object database for Objective-C that supports powerful undo, semantic merging, and real-time collaborative editing".
I never used it but maybe it could help ;)
Cheers
you can convert your database rows to xml and do version control on this textual data.
I believe this is what http://buzzdata.com (http://buzzdata.com/faq/about) was trying to do. Sadly, it closed down. I don't think anyone has picked up the baton. I think it would be a great idea.
You say: "in CSV rather than line vs line comparison, it would do cell vs cell"
You might convert the CSV to JSON with key:value on each line:
Example:
{
Name: Bill,
Title: Doctor
}
{
Name: Super-Ted,
Title: Coder
}
.
.
.
Recently (november 2016), ClusterHQ released a new tool called fli that seem to be promising. I have not tried it yet, but it seems it plays well with many database system (mysql, mongodb, elastic, etc).
Moreover, I do not know if it plays well with GIT and it will certainly be anoying to work with 2 cli toolset, 2 revision system, 2 repository ... for each project. It would be important to manage both data and text with the same system if one wants to keep the essential improvement provided by GIT to projet management and decentralized/collaborative work.
GIT design is fine for data management. Branches, merges, pull requests, etc. and more recent submodules and subworktree are powerful features that radically transformed our day-to-day workflow. The only problem with GIT fot data is that its actual implementation is focused on text files. It track changes with an extensive use diff. That is fine for code or writing projects, but it is clearly not an optimal way to manage big datasets.
A possible improvement within GIT itself that could enable its use for data would be to simply replace everything diff operation applied to a database with something more appropriate for this type of dataformat.
Semantic Web and linked data are made by RDF...
There are a solution at http://aksw.org/Projects/Quit.html
Any kind of RDF: XML, JSON-LD, HTML-RDFa, etc.
Use http://goodtables.okfnlabs.org for continuous validation of tabular data.
When you preserve "allways valid CSV", and no big changes, you can track changes by usual diff offered by usual git.
qri (pronounced "query")
Qri is a distributed dataset version control system built with peer-2-peer data exchange. Peers create datasets, which are stored in versions. Qri peers form a distributed network to exchange information about their datasets, which they transmit between each other over the distributed web.
License is GPLv3
Git is actually quite versatile and you can configure it to handle different types of files differently. Here is an example of how to adapt it to diff CSV files in a more useful way: http://theodi.org/blog/adapting-git-simple-data
The Datahub intiative at MIT ( https://datahub.csail.mit.edu/www/ ) is experimenting in something that has analogies to github, but for data. This is subtly different for pure git since it is hosted activity as well as a versioning and tracking system. It does try to address the versioning and tracking functionality that git provides for code, but in a way that can apply to data.
The development code for Datahub is available in github
gitannex is an extension of Git designed to handle large data files and derived processed data (see also the wiki page). It is being used by major data archives such as OpenNeuro.
https://www.liquidata.co is supposed to be "git for data".
