114

I know you can put CSV or XML in Git already, but there are issues for those wishing to collaborate on a dataset for creation, cleaning it up, pull requests, etc. Are there alternative version control systems that suit data better?

The key improvement I'd want is to Diff/Merge more intelligently. e.g. in CSV rather than line vs line comparison, it would do cell vs cell.

And ordering is usually not significant, e.g. rows in a CSV, whereas Git does care and presents the user with 'conflicts'.

2
  • In CSV, a cell-vs-cell merge is the same as a line-vs-line merge - same result. For Diff, it's different, but you can use git's word-diff to deal with that (output formatting won't be nicely tabular though).
    – naught101
    Aug 14, 2015 at 0:32

21 Answers 21

45

I recently stumbled on this article by the Open Knowledge Foundation regarding the design of a graphical interface to diff tabular data called daff.

It can also be tested and forked on GitHub.

2
  • 2
    The Coopy tools appear to cover all the key issues - intelligent diffs, merge conflicts, basic git integration, merging with CSV to XLS and SQL, web interface. It seems to be the best solution so far, so let's use it and improve it.
    – D Read
    Aug 28, 2013 at 12:13
  • @ paulfitz - a great tool! Aug 29, 2014 at 13:13
43

Basic answer afaik is: "No".

As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures.

Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as suggested by Deer Hunter).

Fundamentally git or hg or a given VCS is built around a diff format and a merge protocol. Get these right and much of the rest follows. The basic 3 options you have are:

  1. Serialize to line-oriented text and use the great tools like git
  2. Identify atomic structure (e.g. document) and apply diff at that level (think CouchDB or standard copy-on-write for RDBMS at row level)
  3. Recording transforms (e.g. Refine)

So far, whilst imperfect (1) beats out anything I've really seen for 2 or 3.

Update: I've written some of this up in much more detail in this post which recommends using simple text-based formats and then using version control like Git - http://blog.okfn.org/2013/07/02/git-and-github-for-data/

Update: there is also the Frictionless Data project which is working to making this kind of thing easier. In particular, the "Core Datasets" part of that is actively working to maintain a set of "core" datasets in version control on Github - see http://data.okfn.org/roadmap/core-datasets and https://github.com/datasets

More info

Here's a long post from 3y ago (my how time flies) on why we would want this and what's difficult about it:

http://blog.okfn.org/2010/07/12/we-need-distributed-revisionversion-control-for-data/

Here's a summary of concepts around versioning for data: http://www.dataprotocols.org/en/latest/revisioning-data.html

The section entitled "diffing and merging" in this post has a summary of the basic options.

4
  • This is excellent analysis of the issues, with some reasonable angles of attack. It is clear that creating any new DCVS is a big piece of work. Piggybacking on git itself (via serialization) seems the best way to get this project started, to try out some of these methods of diff and merge, to see what might be workable. Thanks!
    – D Read
    Jun 11, 2013 at 15:32
  • To avoid inevitable dead links, please post the relevant excerpt from the links in your answer.
    – Kermit
    Jun 12, 2013 at 20:54
  • 3
    Since writing this answer, Rufus has written a longer blog post about "Git (and Github) for Data" at blog.okfn.org/2013/07/02/git-and-github-for-data Jul 4, 2013 at 13:32
  • Rufus' bottom line is to encourage us to simply use git(hub) for CSV (except for large datasets) and says he's been using it with some success on some official datasets he's put on Github. He's highlighted the benefits of collaboration by adding a schema and scripts to data repo with the data. This is all great, however he has pre-cleaned data that goes into git into a good format already - the changesets are mostly addition of rows. I think there is more scope to try out git to "collaborate on a dataset for creation, cleaning it up, pull requests, etc"
    – D Read
    Jul 1, 2014 at 17:04
31

There is a project dat that addresses what you are looking for. It is now in beta, featuring a command line tool with similar design like git. There are also APIs for Javascript (dat-core), Python (datpy) and R (rdat).

Supported input data formats for tabular data are (as of August 2015): CSV, TSV, JSON, and newline delimited JSON. It also supports versioning binary blobs (e.g. image data) as a simple key-value storage.

There is a whitepaper that lists the design goals and workflows that are now working or expected to work with dat. Like git, it already supports distributed versioning with forking and merging.

For trying it out, there is an interactive tutorial at http://try-dat.com/.

Also take a look at "dat jawn: 'Git for Tabular Data'" at https://github.com/CodeForPhilly/jawn .

5
  • 1
    This describes git's difficulties when working with lots of small changes to data. The dat project aims to use CouchDB to do the storage/versioning and provide a GitHub-style web interface for collaboration - all very promising!
    – D Read
    Jul 8, 2013 at 9:28
  • If they reach their design goals, it really could become "the Git for data".
    – ojdo
    Oct 2, 2013 at 14:48
  • 1
    The project funding til Feb 2014 funds a prototype, but a Web GUI and diff/merge are out of scope. So its still a long way from being a collaboration tool. The focus is on syncing data and transforming it at the moment.
    – D Read
    Dec 5, 2013 at 10:49
  • I think that section got deleted.
    – asmeurer
    Jan 29, 2015 at 16:03
  • asmeurer: I updated the post to reflect that.
    – ojdo
    Aug 25, 2015 at 9:08
20

Please note that git has two configuration commands:

git config filter.<driver>.clean
git config filter.<driver>.smudge

The clean filter can be used e.g. to sort all records in a CSV file except the top row, thus making re-ordering irrelevant.

Quoting gitattributes(5) :

A filter driver consists of a clean command and a smudge command, either of which can be left unspecified. Upon checkout, when the smudge command is specified, the command is fed the blob object from its standard input, and its standard output is used to update the worktree file. Similarly, the clean command is used to convert the contents of worktree file upon checkin.

A missing filter driver definition in the config is not an error but makes the filter a no-op passthru.

The content filtering is done to massage the content into a shape that is more convenient for the platform, filesystem, and the user to use. The key phrase here is "more convenient" and not "turning something unusable into usable". In other words, the intent is that if someone unsets the filter driver definition, or does not have the appropriate filter program, the project should still be usable.

1
  • This seems a good way to solve one of the two problems identified.
    – D Read
    Jun 11, 2013 at 15:34
16

Can you provide a more comprehensive list of the pain points you're experiencing with Git? Git works great for data.

In addition to the config flag Deer Hunter mentioned for sorting, you can also teach git to diff on a word-by-word (rather than line-by-line basis), by simply passing the --word-diff flag.

3
  • 4
    As an alternative to git diff --word-diff, you can also use git diff --color-words, which is much prettier :) Jun 11, 2013 at 21:51
  • 1
    Super! This solves the other issue I raised. I'll happily share any other issues I encounter.
    – D Read
    Jun 11, 2013 at 22:09
  • 1
    I've had problems using git with my dataset that has roughly 350 million rows and 10-50 columns, with many missing values that are filled with imputation. I'd like to track which values are imputed and by which version of my software stack. I would then like to be able to fix bugs in the stack and have only the values that are changed be updated. Can you give me advice on how to do this with git?
    – vy32
    Sep 7, 2020 at 15:26
9

Since this question was originally asked, the Dat project has made a lot of progress. Originally conceived by Max Ogden, it now has several other developers working on it.

dat is an open source tool that enables the sharing of large datasets, the goal being a collaboration flow similar to what git offers for source code. As a team we have a bias towards supporting scientific + research data use cases.

The screencast linked from the home page (or here on YouTube) is worth watching if you're interested in this kind of stuff.

1
  • 2
    There are clearly lots of good things going on with dat, but the focus is still data distribution, rather than collaboration. By collaboration I mean allowing different people to make changes or additions to a dataset and then merging them. Git is not just about versioned storage - the branching and merging thing is a pretty essential part of its value. Certainly this functionality would be a good fit with dat.
    – D Read
    May 27, 2014 at 10:06
9

Another open-source one that has popped up is DVC

https://github.com/iterative/dvc

Data Science Version Control or DVC is an open-source tool for data science and machine learning projects. With a simple and flexible Git-like architecture and interface

DVC is compatible with Git for storing code and the dependency graph (DAG), but not data files cache. To store and share data files cache DVC supports remotes - any cloud (S3, Azure, Google Cloud, etc) or any on-premise network storage (via SSH, for example).

enter image description here

5

Yes there is one, it is called noms

https://github.com/attic-labs/noms

Noms is a decentralized database philosophically descendant from the Git version control system.

1
4

Git's diff-ing and merge-ing functionality can be extended using custom drivers. I just wrote a custom merging driver for json files that merges based on location in the tree instead of on a line-by-line basis. It's only 96 lines of coffee-script (with comments): https://gist.github.com/jphaas/ad7823b3469aac112a52

I can imagine doing something similar for CSV files. The algorithm should be even simpler since CSV isn't a recursive format.

Given how easy it is to extend Git like this, I suggest going down the custom driver path rather than trying to massage the data into a line-by-line format.

1
  • This is exactly the mechanism that Coopy uses to integrate with Git. It provides intelligent diff & merge for CSVs: see share.find.coop/doc/tutorial_git.html - really quite impressive.
    – D Read
    Jun 27, 2014 at 13:21
4

This topic was also discussed sometime ago at answers.semanticweb.com, see

Here are also some links to some semantic versioning approaches:

7
  • 5
    Are there highlights from any of those pages you could add to this answer as a safeguard against link rot? Jul 4, 2013 at 13:36
  • dunno, I personally preselected this collection of links for you, because I thought they are dealing with the topic your are discussing in this thread. I also think that there are already complete good answers in it (in the threads from answers.semanticweb.com). Furthermore, since answers.semanticweb.com is a board with many intersecting topics of this board, I hope that it wouldn't be a problem to just link specific threads and don't repeat everything again on this board here. I'm often also short in time. So I hope that just sometimes posting some referring links might be okay for you, or?
    – zazi
    Jul 5, 2013 at 8:25
  • I totally understand—I've been in the same situation myself. It's better to put the links than nothing. But when possible, the community considers direct answers preferable to links. Jul 5, 2013 at 13:14
  • 1
    @zazi trust has nothing to do with it, no offense. your first two links succumbed to linkrot...
    – albert
    Mar 3, 2017 at 20:47
  • 1
    thx for fixing the links @albert ;)
    – zazi
    Aug 11, 2017 at 12:20
3

I don't know if you are still looking for a git for data but there is Core Object that present themselves as: "a version-controlled object database for Objective-C that supports powerful undo, semantic merging, and real-time collaborative editing".

I never used it but maybe it could help ;)

Cheers

1
  • Thanks. Writing a DVCS with hackable diffing and merging is an excellent start. The semantic merging is pretty neat - the video youtube.com/watch?v=V8hvq79YpdM shows merging changes in a graphics editor, so no doubt one could be written for CSV changes. It would also need packaging for command-line use.
    – D Read
    Jun 4, 2014 at 14:10
3

you can convert your database rows to xml and do version control on this textual data.

1
  • XML appears to helps diff by putting each cell value on a different line. However automatic merge sounds a nightmare. Imagine it defaults to comparing 3 lines either-side - if those 3 lines are full of verbose XML boilerer-plate, and perhaps 5 adjacent values - it's not a lot to keep the right values changes in the right records. Git just feels like the wrong tool for this job.
    – D Read
    Jun 23, 2014 at 15:53
2

I believe this is what http://buzzdata.com (http://buzzdata.com/faq/about) was trying to do. Sadly, it closed down. I don't think anyone has picked up the baton. I think it would be a great idea.

2

You say: "in CSV rather than line vs line comparison, it would do cell vs cell"

You might convert the CSV to JSON with key:value on each line:

Example:

{
Name: Bill,
Title: Doctor
}
{
Name: Super-Ted,
Title: Coder
}
.
.
.
1
  • JSON is popular for APIs and Javascript. Collaborating with data is more about Excel, python, databases etc. for which JSON is generally less convenient than CSV. And generally it would be a challenge to get all collaborating software to insert the carriage returns automatically and consistently to use this method. So yes an option, but rather limited for widespread data collaboration.
    – D Read
    Sep 2, 2013 at 9:29
2

Recently (november 2016), ClusterHQ released a new tool called fli that seem to be promising. I have not tried it yet, but it seems it plays well with many database system (mysql, mongodb, elastic, etc).

Moreover, I do not know if it plays well with GIT and it will certainly be anoying to work with 2 cli toolset, 2 revision system, 2 repository ... for each project. It would be important to manage both data and text with the same system if one wants to keep the essential improvement provided by GIT to projet management and decentralized/collaborative work.

GIT design is fine for data management. Branches, merges, pull requests, etc. and more recent submodules and subworktree are powerful features that radically transformed our day-to-day workflow. The only problem with GIT fot data is that its actual implementation is focused on text files. It track changes with an extensive use diff. That is fine for code or writing projects, but it is clearly not an optimal way to manage big datasets.

A possible improvement within GIT itself that could enable its use for data would be to simply replace everything diff operation applied to a database with something more appropriate for this type of dataformat.

1
  • I reviewed your edit proposal Jan Doggen. But has I am new here, I do not have that privilege and it was lost... I'll rewrite it this evening. Briefly, I do not accept that revision because it is too drastic. There is somme essential idea that I would like to keep. But I keep your comment in mind and completely rewrite my answer.
    – jvtrudel
    Mar 3, 2017 at 21:09
2

For RDF-data: Quit

Semantic Web and linked data are made by RDF...

There are a solution at http://aksw.org/Projects/Quit.html

Any kind of RDF: XML, JSON-LD, HTML-RDFa, etc.

For CSV data: Goodtables

Use http://goodtables.okfnlabs.org for continuous validation of tabular data.

When you preserve "allways valid CSV", and no big changes, you can track changes by usual diff offered by usual git.

2

qri (pronounced "query")

Qri is a distributed dataset version control system built with peer-2-peer data exchange. Peers create datasets, which are stored in versions. Qri peers form a distributed network to exchange information about their datasets, which they transmit between each other over the distributed web.

1

Git is actually quite versatile and you can configure it to handle different types of files differently. Here is an example of how to adapt it to diff CSV files in a more useful way: http://theodi.org/blog/adapting-git-simple-data

2
  • 1
    Looks like James used nearly all the tips from contributors to this page and then added a couple of his own. The addition of wordRegex seems a mildly useful addition to the --word-diff approach. And his mod for visual diffs in a github-like site is a promising direction. However there still seem to be serious shortcomings with git for data. For example, merging when someone's added a column is still going to be horrendous, for example.
    – D Read
    Aug 20, 2013 at 9:40
  • James' latest work handily adds a browser plugin to display CSV diffs better in Github: theodi.org/blog/csvhub-github-diffs-for-csv-files (tip: Ulrich)
    – D Read
    Jun 4, 2014 at 19:56
1

The Datahub intiative at MIT ( https://datahub.csail.mit.edu/www/ ) is experimenting in something that has analogies to github, but for data. This is subtly different for pure git since it is hosted activity as well as a versioning and tracking system. It does try to address the versioning and tracking functionality that git provides for code, but in a way that can apply to data.

The development code for Datahub is available in github

1
  • Another option that has analogies to Github (with git) is Dolthub (with dolt).
    – Asclepius
    Jul 9, 2023 at 14:42
1

gitannex is an extension of Git designed to handle large data files and derived processed data (see also the wiki page). It is being used by major data archives such as OpenNeuro.

2
  • 1
    Hi! Thanks for the name of this tool. Please supply details of how it addresses the challenges in the question e.g. versioning, diffing & merging.
    – D Read
    Sep 26, 2018 at 6:44
  • And add a link, please.
    – user4293
    Sep 26, 2018 at 9:49
1

https://www.liquidata.co is supposed to be "git for data".

2
  • "Dolt versions tables" is all that their website says. Apparently in private beta. Looks like a start-up. Let's hope it will meet the needs of people collaborating on datasets, such as being open source and do a good job of diffs and merges.
    – D Read
    Jan 13, 2019 at 11:05
  • @DRead: They posted here at opendata, but apparently the post was deleted, because moderators considered it advertisement. You can write to them and ask for access. I did.
    – Make42
    Jan 14, 2019 at 9:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.