I'm looking for messy data files for practice/testing/training in data cleaning. Ideally, someone knows of a collection like there are for Machine Learning and other domains. But individual bad data sets would be useful, too. By "bad" I meaning containing things like misspellings, miscodings, wrong level of detail, ...
I collected some answer from friends on twitter, they suggested:
- MG-RAST: http://metagenomics.anl.gov/
- Interaction web database: http://www.nceas.ucsb.edu/interactionweb/ - just a collection of csv files basically, same general format but all from diff researchers
- Interesting dataset of 2127 articles that the Wellcome trust paid for in Europe for open access article fees to publishers
- lots of messy data see this blog post: http://biomickwatson.wordpress.com/2014/03/25/biologists-this-is-why-bioinformaticians-hate-you/
- the data http://figshare.com/articles/Wellcome_Trust_APC_spend_2012_13_data_file/963054
If you want messy data to test cleaning features, maybe you can start with clean data and then apply some minor changes here and there to corrupt your original data.
This is the strategy I followed to test a screening system that need to detect some words (the clean data) in SWIFT messages even if they occur with some minor typos.
I used GEDIS Studio to do so and here how I proceeded :
The idea was to start with a collection of clean data representing personal data (first name, last name, birth date, address, etc.). I used a generator available in GEDIS (the one named "Personnal Data" in the shared workspace).
Then produce a dataset of personnal data in CSV format.
Then I created another generator using a specific transformation rule named "The Word Jumbler" to apply various minor changes on the selected columns of the source dataset (basically names).
I apply the second generator with input read from the dataset of personal data
The available jumbling effects in the Jumbler rule are
- character swapping, where you can configure the distance between the swapped positions
- character insertion, where you can configure the number of character to insert and the alphabet of character you can insert
- character change where you can configure the alphabet of changeable characters, the alphabet of characters you can insert
Obviously, you can mix several transformation at a time for the same source value.
The VERY valuable feature is that you can also produce a table with the original data, the corrupted data and the type of corruption applied to make sure that you detect all dirty data, only dirty data and if you miss ones then what were the type of corruption you missed.
Cool no ?
If you want to check that mechanism on a simple use case, you can register an account on GEDIS Studio at http://www.data-generator.com (its FREE) and import the "Testing" project from the shared workspace. Then go to the Production tab and launch generation of the file named "String Mutations.txt" it is based on the "String Mutations" generator.
In the following picture you can see the result of this sample applied over the OFAC list.
Do not hesitate to contact me if you have trouble with this.
Hope this helps
I started an article back in Dec (haven't finished) about handling bad data in government datasets. Perhaps there is enough info to seed bad data into an existing dataset at a predetermined distribution:
Dec. 18, 2013 By Andrew Ferlitsch
Problem: How we deal with bad data in government datasets.
While working with datasets from various government agencies, both US and abroad, as well as international organizations, we encounter what we call bad data. Bad data effects the value of the dataset and may cause problems in parsing the data, importation into databases, and indexing and search. Bad data generally falls into the following categories:
Wrong character encoding of dataset - results in garbled characters. Misalignment of data - results in breaking non-adaptable parsing methods. Errors in metadata descriptions - results in misinterpretation of fields. Non-standard or esoteric representation of primary keys - results in problematic indexing of records. Inconsistencies in delimination and use of punctuation with fields - results in problematic decomposition of the data. Obsolete or historic data not identified accordingly - results in invalid (erroneous) records in database.
I'm a big fan of twitter data because with the 140 characters of text per tweet comes a network and enourmous amounts of meta-data, including many incorrect computer-assigned tags (i.e. tweet language). What can you do with 140 characters and some meta-data: A recent example of researchers determining a user's location based on non-geotagged tweets (link).
I've previously posted two methods for collecting tweet data.
If you run the live stream, you can collect your own (you'll get more tweets than you can deal with).
Tweets from the live stream can come without any language filtering, and each tweet is assigned a language
tweet[u'lang'] in addition to the user's language
tweet[u'user'][u'lang']. You'll find that many of these auto-assignments are incorrect. For example, here is a graph of the languages of my tweets, which includes Slovene and Tagalog (link to tweet).
Another option is to search for tweets.
If you want to 'search' for all tweets, you can use
lang:enas your search parameter. See here. Then you can loop over desired langauges.
In the tweets returned with
lang:en you'll find many misspellings, abbreviations, etc.