I know you can put CSV or XML in Git already, but there are issues for those wishing to collaborate on a dataset for creation, cleaning it up, pull requests, etc. Are there alternative version control systems that suit data better?

The key improvement I'd want is to Diff/Merge more intelligently. e.g. in CSV rather than line vs line comparison, it would do cell vs cell.

And ordering is usually not significant, e.g. rows in a CSV, whereas Git does care and presents the user with 'conflicts'.

  • In CSV, a cell-vs-cell merge is the same as a line-vs-line merge - same result. For Diff, it's different, but you can use git's word-diff to deal with that (output formatting won't be nicely tabular though). – naught101 Aug 14 '15 at 0:32
  • 1
    Look at noms. – Ahmed Fasih Aug 25 '16 at 0:42
  • noms looks promising - please add details in an answer – D Read Aug 25 '16 at 6:07

18 Answers 18

up vote 31 down vote accepted

I recently stumbled on this article by the Open Knowledge Foundation regarding the design of a graphical interface to diff tabular data called daff.

It can also be tested and forked on GitHub.

  • 2
    The Coopy tools appear to cover all the key issues - intelligent diffs, merge conflicts, basic git integration, merging with CSV to XLS and SQL, web interface. It seems to be the best solution so far, so let's use it and improve it. – D Read Aug 28 '13 at 12:13
  • @ paulfitz - a great tool! – Deer Hunter Aug 29 '14 at 13:13

Basic answer afaik is: "No".

As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures.

Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as suggested by Deer Hunter).

Fundamentally git or hg or a given VCS is built around a diff format and a merge protocol. Get these right and much of the rest follows. The basic 3 options you have are:

  1. Serialize to line-oriented text and use the great tools like git
  2. Identify atomic structure (e.g. document) and apply diff at that level (think CouchDB or standard copy-on-write for RDBMS at row level)
  3. Recording transforms (e.g. Refine)

So far, whilst imperfect (1) beats out anything I've really seen for 2 or 3.

Update: I've written some of this up in much more detail in this post which recommends using simple text-based formats and then using version control like Git - http://blog.okfn.org/2013/07/02/git-and-github-for-data/

Update: there is also the Frictionless Data project which is working to making this kind of thing easier. In particular, the "Core Datasets" part of that is actively working to maintain a set of "core" datasets in version control on Github - see http://data.okfn.org/roadmap/core-datasets and https://github.com/datasets

More info

Here's a long post from 3y ago (my how time flies) on why we would want this and what's difficult about it:

http://blog.okfn.org/2010/07/12/we-need-distributed-revisionversion-control-for-data/

Here's a summary of concepts around versioning for data: http://www.dataprotocols.org/en/latest/revisioning-data.html

The section entitled "diffing and merging" in this post has a summary of the basic options.

  • This is excellent analysis of the issues, with some reasonable angles of attack. It is clear that creating any new DCVS is a big piece of work. Piggybacking on git itself (via serialization) seems the best way to get this project started, to try out some of these methods of diff and merge, to see what might be workable. Thanks! – D Read Jun 11 '13 at 15:32
  • To avoid inevitable dead links, please post the relevant excerpt from the links in your answer. – Kermit Jun 12 '13 at 20:54
  • 2
    Since writing this answer, Rufus has written a longer blog post about "Git (and Github) for Data" at blog.okfn.org/2013/07/02/git-and-github-for-data – Joe Germuska Jul 4 '13 at 13:32
  • Rufus' bottom line is to encourage us to simply use git(hub) for CSV (except for large datasets) and says he's been using it with some success on some official datasets he's put on Github. He's highlighted the benefits of collaboration by adding a schema and scripts to data repo with the data. This is all great, however he has pre-cleaned data that goes into git into a good format already - the changesets are mostly addition of rows. I think there is more scope to try out git to "collaborate on a dataset for creation, cleaning it up, pull requests, etc" – D Read Jul 1 '14 at 17:04

There is a project dat that addresses what you are looking for. It is now in beta, featuring a command line tool with similar design like git. There are also APIs for Javascript (dat-core), Python (datpy) and R (rdat).

Supported input data formats for tabular data are (as of August 2015): CSV, TSV, JSON, and newline delimited JSON. It also supports versioning binary blobs (e.g. image data) as a simple key-value storage.

There is a whitepaper that lists the design goals and workflows that are now working or expected to work with dat. Like git, it already supports distributed versioning with forking and merging.

For trying it out, there is an interactive tutorial at http://try-dat.com/.

Also take a look at "dat jawn: 'Git for Tabular Data'" at https://github.com/CodeForPhilly/jawn .

  • 1
    This describes git's difficulties when working with lots of small changes to data. The dat project aims to use CouchDB to do the storage/versioning and provide a GitHub-style web interface for collaboration - all very promising! – D Read Jul 8 '13 at 9:28
  • If they reach their design goals, it really could become "the Git for data". – ojdo Oct 2 '13 at 14:48
  • 1
    The project funding til Feb 2014 funds a prototype, but a Web GUI and diff/merge are out of scope. So its still a long way from being a collaboration tool. The focus is on syncing data and transforming it at the moment. – D Read Dec 5 '13 at 10:49
  • I think that section got deleted. – asmeurer Jan 29 '15 at 16:03
  • asmeurer: I updated the post to reflect that. – ojdo Aug 25 '15 at 9:08

Please note that git has two configuration commands:

git config filter.<driver>.clean
git config filter.<driver>.smudge

The clean filter can be used e.g. to sort all records in a CSV file except the top row, thus making re-ordering irrelevant.

Quoting gitattributes(5) :

A filter driver consists of a clean command and a smudge command, either of which can be left unspecified. Upon checkout, when the smudge command is specified, the command is fed the blob object from its standard input, and its standard output is used to update the worktree file. Similarly, the clean command is used to convert the contents of worktree file upon checkin.

A missing filter driver definition in the config is not an error but makes the filter a no-op passthru.

The content filtering is done to massage the content into a shape that is more convenient for the platform, filesystem, and the user to use. The key phrase here is "more convenient" and not "turning something unusable into usable". In other words, the intent is that if someone unsets the filter driver definition, or does not have the appropriate filter program, the project should still be usable.

  • This seems a good way to solve one of the two problems identified. – D Read Jun 11 '13 at 15:34

Can you provide a more comprehensive list of the pain points you're experiencing with Git? Git works great for data.

In addition to the config flag Deer Hunter mentioned for sorting, you can also teach git to diff on a word-by-word (rather than line-by-line basis), by simply passing the --word-diff flag.

  • 4
    As an alternative to git diff --word-diff, you can also use git diff --color-words, which is much prettier :) – Patrick Hoefler Jun 11 '13 at 21:51
  • 1
    Super! This solves the other issue I raised. I'll happily share any other issues I encounter. – D Read Jun 11 '13 at 22:09

Since this question was originally asked, the Dat project has made a lot of progress. Originally conceived by Max Ogden, it now has several other developers working on it.

dat is an open source tool that enables the sharing of large datasets, the goal being a collaboration flow similar to what git offers for source code. As a team we have a bias towards supporting scientific + research data use cases.

The screencast linked from the home page (or here on YouTube) is worth watching if you're interested in this kind of stuff.

  • 2
    There are clearly lots of good things going on with dat, but the focus is still data distribution, rather than collaboration. By collaboration I mean allowing different people to make changes or additions to a dataset and then merging them. Git is not just about versioned storage - the branching and merging thing is a pretty essential part of its value. Certainly this functionality would be a good fit with dat. – D Read May 27 '14 at 10:06

Git's diff-ing and merge-ing functionality can be extended using custom drivers. I just wrote a custom merging driver for json files that merges based on location in the tree instead of on a line-by-line basis. It's only 96 lines of coffee-script (with comments): https://gist.github.com/jphaas/ad7823b3469aac112a52

I can imagine doing something similar for CSV files. The algorithm should be even simpler since CSV isn't a recursive format.

Given how easy it is to extend Git like this, I suggest going down the custom driver path rather than trying to massage the data into a line-by-line format.

  • This is exactly the mechanism that Coopy uses to integrate with Git. It provides intelligent diff & merge for CSVs: see share.find.coop/doc/tutorial_git.html - really quite impressive. – D Read Jun 27 '14 at 13:21

Yes there is one, it is called noms

https://github.com/attic-labs/noms

This topic was also discussed sometime ago at answers.semanticweb.com, see

Here are also some links to some semantic versioning approaches:

  • 5
    Are there highlights from any of those pages you could add to this answer as a safeguard against link rot? – Joe Germuska Jul 4 '13 at 13:36
  • dunno, I personally preselected this collection of links for you, because I thought they are dealing with the topic your are discussing in this thread. I also think that there are already complete good answers in it (in the threads from answers.semanticweb.com). Furthermore, since answers.semanticweb.com is a board with many intersecting topics of this board, I hope that it wouldn't be a problem to just link specific threads and don't repeat everything again on this board here. I'm often also short in time. So I hope that just sometimes posting some referring links might be okay for you, or? – zazi Jul 5 '13 at 8:25
  • I totally understand—I've been in the same situation myself. It's better to put the links than nothing. But when possible, the community considers direct answers preferable to links. – Joe Germuska Jul 5 '13 at 13:14
  • 1
    @zazi trust has nothing to do with it, no offense. your first two links succumbed to linkrot... – albert Mar 3 '17 at 20:47
  • 1
    thx for fixing the links @albert ;) – zazi Aug 11 '17 at 12:20

I don't know if you are still looking for a git for data but there is Core Object that present themselves as: "a version-controlled object database for Objective-C that supports powerful undo, semantic merging, and real-time collaborative editing".

I never used it but maybe it could help ;)

Cheers

  • Thanks. Writing a DVCS with hackable diffing and merging is an excellent start. The semantic merging is pretty neat - the video youtube.com/watch?v=V8hvq79YpdM shows merging changes in a graphics editor, so no doubt one could be written for CSV changes. It would also need packaging for command-line use. – D Read Jun 4 '14 at 14:10

you can convert your database rows to xml and do version control on this textual data.

  • XML appears to helps diff by putting each cell value on a different line. However automatic merge sounds a nightmare. Imagine it defaults to comparing 3 lines either-side - if those 3 lines are full of verbose XML boilerer-plate, and perhaps 5 adjacent values - it's not a lot to keep the right values changes in the right records. Git just feels like the wrong tool for this job. – D Read Jun 23 '14 at 15:53

You say: "in CSV rather than line vs line comparison, it would do cell vs cell"

You might convert the CSV to JSON with key:value on each line:

Example:

{
Name: Bill,
Title: Doctor
}
{
Name: Super-Ted,
Title: Coder
}
.
.
.
  • JSON is popular for APIs and Javascript. Collaborating with data is more about Excel, python, databases etc. for which JSON is generally less convenient than CSV. And generally it would be a challenge to get all collaborating software to insert the carriage returns automatically and consistently to use this method. So yes an option, but rather limited for widespread data collaboration. – D Read Sep 2 '13 at 9:29

Recently (november 2016), ClusterHQ released a new tool called fli that seem to be promising. I have not tried it yet, but it seems it plays well with many database system (mysql, mongodb, elastic, etc).

Moreover, I do not know if it plays well with GIT and it will certainly be anoying to work with 2 cli toolset, 2 revision system, 2 repository ... for each project. It would be important to manage both data and text with the same system if one wants to keep the essential improvement provided by GIT to projet management and decentralized/collaborative work.

GIT design is fine for data management. Branches, merges, pull requests, etc. and more recent submodules and subworktree are powerful features that radically transformed our day-to-day workflow. The only problem with GIT fot data is that its actual implementation is focused on text files. It track changes with an extensive use diff. That is fine for code or writing projects, but it is clearly not an optimal way to manage big datasets.

A possible improvement within GIT itself that could enable its use for data would be to simply replace everything diff operation applied to a database with something more appropriate for this type of dataformat.

  • I reviewed your edit proposal Jan Doggen. But has I am new here, I do not have that privilege and it was lost... I'll rewrite it this evening. Briefly, I do not accept that revision because it is too drastic. There is somme essential idea that I would like to keep. But I keep your comment in mind and completely rewrite my answer. – jvtrudel Mar 3 '17 at 21:09

I believe this is what http://buzzdata.com (http://buzzdata.com/faq/about) was trying to do. Sadly, it closed down. I don't think anyone has picked up the baton. I think it would be a great idea.

Git is actually quite versatile and you can configure it to handle different types of files differently. Here is an example of how to adapt it to diff CSV files in a more useful way: http://theodi.org/blog/adapting-git-simple-data

  • 1
    Looks like James used nearly all the tips from contributors to this page and then added a couple of his own. The addition of wordRegex seems a mildly useful addition to the --word-diff approach. And his mod for visual diffs in a github-like site is a promising direction. However there still seem to be serious shortcomings with git for data. For example, merging when someone's added a column is still going to be horrendous, for example. – D Read Aug 20 '13 at 9:40
  • James' latest work handily adds a browser plugin to display CSV diffs better in Github: theodi.org/blog/csvhub-github-diffs-for-csv-files (tip: Ulrich) – D Read Jun 4 '14 at 19:56

I might be late to the party but datachili.com does exactly this. It's a collaborative platform to cleanse/prepare/normalize data (and this can be big-data). So basically multiple users working on the same data in real-time. Let me know if you're interested and I can probably setup demo accounts.

Disclosure : I work for datachili.com

  • I don't know if demo accounts are necessary, but it might be a good idea to put together a screencast / video of some of the features. Although I'm not normally a fan of watching videos that can be explained in text, it's often nice to see what someone's workflow would be to see how difficult it is to use a tool. – Joe May 25 '16 at 20:49
  • @Joe We're setting that up within the next 2 weeks. If its urgent, feel free to get in touch with me at dhruv@datachili.com – Dhruv Gairola May 25 '16 at 21:35
  • This answer doesn't address the key requirements of the question - pull requests, diff & merge. There is only marketing info on the single webpage so far - almost zero information apart from a promise of data tools which are multi-user. – D Read Jun 8 '16 at 11:19
  • @DRead We're working on improving/redesigning our landing page so that it's clear what our platform does (it does what this question describes apart from cell-diffs). Feel free to get in touch with me for a demo in the meantime. – Dhruv Gairola Jun 8 '16 at 15:13

The Datahub intiative at MIT ( https://datahub.csail.mit.edu/www/ ) is experimenting in something that has analogies to github, but for data. This is subtly different for pure git since it is hosted activity as well as a versioning and tracking system. It does try to address the versioning and tracking functionality that git provides for code, but in a way that can apply to data.

The development code for Datahub is available in github

For RDF-data: Quit

Semantic Web and linked data are made by RDF...

There are a solution at http://aksw.org/Projects/Quit.html

Any kind of RDF: XML, JSON-LD, HTML-RDFa, etc.

For CSV data: Goodtables

Use http://goodtables.okfnlabs.org for continuous validation of tabular data.

When you preserve "allways valid CSV", and no big changes, you can track changes by usual diff offered by usual git.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.