How do I merge similar datasets, not just once but on an ongoing basis? This seems like a reasonably common problem so I'd be interested if there is a tool out there aimed at this problem, as a guided workflow. Or if any general purpose data tools might suit.
The exact problem I'm trying to solve is to curate a dataset of UK public bodies that is based on a number of existing datasets that change over time. The sorts of issues are:
- The names of the bodies vary subtly between the datasets
- The datasets all get updates which I'd like to add in regularly.
- There are thousands of bodies, so plenty of automation is required to match them, yet manual matching is essential too.
- Once it is established that an item needs copying into my dataset, the columns of the source datasets (name, abbreviation, website, etc.) will need to be mapped to my dataset's columns. I might want to do some manual tidying of the values, and not have this overwritten on the next time I 'run' the tool.
I'm very keen to hear other ideas, as the idea I've come up with seems a little esoteric:
- Use crowd-sourcing to match up public body names using Nomenklatura.
- Write a python script to take each data source, put in the standardized names, change the columns to a standard structure and save the output as CSV.
- Keep each CSV in git as the same 'file' but on different git branches. This allows me to merge changes into the master manually, using visual git tools. Git keeps a full record of changes in each dataset and the merge.