Skip to main content
update with subsequent activity
Source Link
Rufus Pollock
  • 1.6k
  • 1
  • 9
  • 9

Basic answer afaik is: "No".

As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures.

Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as suggested by Deer Hunter).

Fundamentally git or hg or a given VCS is built around a diff format and a merge protocol. Get these right and much of the rest follows. The basic 3 options you have are:

  1. Serialize to line-oriented text and use the great tools like git
  2. Identify atomic structure (e.g. document) and apply diff at that level (think CouchDB or standard copy-on-write for RDBMS at row level)
  3. Recording transforms (e.g. Refine)

So far, whilst imperfect (1) beats out anything I've really seen for 2 or 3.

Update: I've written some of this up in much more detail in this post which recommends using simple text-based formats and then using version control like Git - http://blog.okfn.org/2013/07/02/git-and-github-for-data/

Update: there is also the Frictionless Data project which is working to making this kind of thing easier. In particular, the "Core Datasets" part of that is actively working to maintain a set of "core" datasets in version control on Github - see http://data.okfn.org/roadmap/core-datasets and https://github.com/datasets

More info

Here's a long post from 3y ago (my how time flies) on why we would want this and what's difficult about it:

http://blog.okfn.org/2010/07/12/we-need-distributed-revisionversion-control-for-data/

Here's a summary of concepts around versioning for data: http://www.dataprotocols.org/en/latest/revisioning-data.html

The section entitled "diffing and merging" in this post has a summary of the basic options.

Basic answer afaik is: "No".

As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures.

Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as suggested by Deer Hunter).

Fundamentally git or hg or a given VCS is built around a diff format and a merge protocol. Get these right and much of the rest follows. The basic 3 options you have are:

  1. Serialize to line-oriented text and use the great tools like git
  2. Identify atomic structure (e.g. document) and apply diff at that level (think CouchDB or standard copy-on-write for RDBMS at row level)
  3. Recording transforms (e.g. Refine)

So far, whilst imperfect (1) beats out anything I've really seen for 2 or 3.

More info

Here's a long post from 3y ago (my how time flies) on why we would want this and what's difficult about it:

http://blog.okfn.org/2010/07/12/we-need-distributed-revisionversion-control-for-data/

Here's a summary of concepts around versioning for data: http://www.dataprotocols.org/en/latest/revisioning-data.html

The section entitled "diffing and merging" in this post has a summary of the basic options.

Basic answer afaik is: "No".

As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures.

Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as suggested by Deer Hunter).

Fundamentally git or hg or a given VCS is built around a diff format and a merge protocol. Get these right and much of the rest follows. The basic 3 options you have are:

  1. Serialize to line-oriented text and use the great tools like git
  2. Identify atomic structure (e.g. document) and apply diff at that level (think CouchDB or standard copy-on-write for RDBMS at row level)
  3. Recording transforms (e.g. Refine)

So far, whilst imperfect (1) beats out anything I've really seen for 2 or 3.

Update: I've written some of this up in much more detail in this post which recommends using simple text-based formats and then using version control like Git - http://blog.okfn.org/2013/07/02/git-and-github-for-data/

Update: there is also the Frictionless Data project which is working to making this kind of thing easier. In particular, the "Core Datasets" part of that is actively working to maintain a set of "core" datasets in version control on Github - see http://data.okfn.org/roadmap/core-datasets and https://github.com/datasets

More info

Here's a long post from 3y ago (my how time flies) on why we would want this and what's difficult about it:

http://blog.okfn.org/2010/07/12/we-need-distributed-revisionversion-control-for-data/

Here's a summary of concepts around versioning for data: http://www.dataprotocols.org/en/latest/revisioning-data.html

The section entitled "diffing and merging" in this post has a summary of the basic options.

Source Link
Rufus Pollock
  • 1.6k
  • 1
  • 9
  • 9

Basic answer afaik is: "No".

As a first point I'd say that "data" is very broad - much broader than if you said "code". There are all kinds of data in all kinds of different structures.

Focusing down on just tabular data would make it more promising but the basic answer, at present, is that the Git for data is Git (perhaps with a bit of config tweaking as suggested by Deer Hunter).

Fundamentally git or hg or a given VCS is built around a diff format and a merge protocol. Get these right and much of the rest follows. The basic 3 options you have are:

  1. Serialize to line-oriented text and use the great tools like git
  2. Identify atomic structure (e.g. document) and apply diff at that level (think CouchDB or standard copy-on-write for RDBMS at row level)
  3. Recording transforms (e.g. Refine)

So far, whilst imperfect (1) beats out anything I've really seen for 2 or 3.

More info

Here's a long post from 3y ago (my how time flies) on why we would want this and what's difficult about it:

http://blog.okfn.org/2010/07/12/we-need-distributed-revisionversion-control-for-data/

Here's a summary of concepts around versioning for data: http://www.dataprotocols.org/en/latest/revisioning-data.html

The section entitled "diffing and merging" in this post has a summary of the basic options.