5

We are looking for a data set or data sets to test record linkage/identity resolution. The target problem is matching up customers to people on various watch lists. We want to test the basic stuff that everyone does like edit distance, Jaccard similiarity, and so on. We also need to test some of our own heuristics. And we might look at locality sensitive hashing and perhaps clustering.

First we need some some data to test efficacy of different approaches. I am looking for labelled data or something with some ground truth so I can produce a confusion matrix of true/false positive/negatives.

I don't want to turn this into too much of a "what's your favorite data set" question, but given what I've written above, what data sets can the community recommend to me?

Thanks.

Update

Changed name of the question and description to better reflect the problem space (with thanks to Mark Silverberg).

Improve this question
4
  • 1
    The dataset you ask contains personal details and I am not sure that you will find something like this as Open Data. Maybe someone else has already seen a similar one.
    – Tasos
    Jul 31, 2014 at 22:55
  • Anonymized or synthetic data would be fine.
    – ahoffer
    Jul 31, 2014 at 22:59
  • @ahoffer How does what you're looking to do differ from generic database record matching (also known as record linkage -- see en.wikipedia.org/wiki/Record_linkage)?
    – Mark Silverberg
    Aug 1, 2014 at 12:50
  • @Mark. I just scanned the Wikipedia article. It looks like record linkage is a very good description of where I'm headed. The section Identity resolution is especially apropos. We expect to have a mix of structured data from an relational DB, semi-structured data from watch lists, and unstructured text from Web searches.
    – ahoffer
    Aug 1, 2014 at 17:32

3 Answers 3

Reset to default
3

If you are looking to generate large data sets and don't mind putting in a bit of work you can use the data set generator in febrl from the Australian National University (Project at http://sourceforge.net/projects/febrl/ and documentation for dataset generator http://cs.anu.edu.au/~Peter.Christen/Febrl/febrl-0.3/febrldoc-0.3/node70.html).

It requires that you give it a dictionary of terms for each field with frequencies and possible misspellings etc and then input the probabilities that this row has an error/is a duplicate etc. There are some dictionaries bundled but they are Australian based but they give you an idea of how to create your own.

It then generates a file with the original record and the duplicates which it identifies for you. Could be useful?

Improve this answer
3

The classic, academic data sets are Restaurants, Cora, Citeseer, and DBLP. You can get them from from the "Repository of Information on Duplicate Detection, Record Linkage, and Identity Uncertainty"

The dedupe project also has some data sets you can use for evaluation: https://github.com/datamade/dedupe-examples

Improve this answer
2

Take a look at Benchmark datasets for entity resolution as well.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.