7

I'm looking for messy data files for practice/testing/training in data cleaning. Ideally, someone knows of a collection like there are for Machine Learning and other domains. But individual bad data sets would be useful, too. By "bad" I meaning containing things like misspellings, miscodings, wrong level of detail, ...

  • 6
    Any gov't dataset taken at random will be messy by default :) – Deer Hunter Mar 27 '14 at 4:11
  • 1
    Just about everything on Wikipedia or Wiktionary would fit this criteria. From article contents to formatting to infoboxes to category structure/hierarchy. – hippietrail Mar 30 '14 at 4:28
  • Some of the worst offendors that I've found are from what we call 'event catalogs' and 'feature catalogs' that have been manually maintained. I even gave a talk on the subject ~6 years ago. (turn on presenter notes for it to make any sense) – Joe Apr 3 '14 at 20:52
  • Thanks @Joe. Looks promising but most of the links don't work. Are there newer URLs? sdac.virtualsolar.org is working (slowly), but the data seems pretty clean (its image data may be great for another project though!) – xan Apr 4 '14 at 13:07
  • @xan : some of the projects I've gotten to clean up their data, so they shouldn't be as bad as they were when I first made the presentation. The NOAA SEP list still has some quirks, as does the CDAW Type II Radio Burst list due to its form of minimiation. (note the CDAW URL isn't loading reliably from my laptop, but is from my desktop) – Joe Apr 7 '14 at 14:46
5

I collected some answer from friends on twitter, they suggested:

3

If you want messy data to test cleaning features, maybe you can start with clean data and then apply some minor changes here and there to corrupt your original data.

This is the strategy I followed to test a screening system that need to detect some words (the clean data) in SWIFT messages even if they occur with some minor typos.

I used GEDIS Studio to do so and here how I proceeded :

  • The idea was to start with a collection of clean data representing personal data (first name, last name, birth date, address, etc.). I used a generator available in GEDIS (the one named "Personnal Data" in the shared workspace).

  • Then produce a dataset of personnal data in CSV format.

  • Then I created another generator using a specific transformation rule named "The Word Jumbler" to apply various minor changes on the selected columns of the source dataset (basically names).

  • I apply the second generator with input read from the dataset of personal data

enter image description here

The available jumbling effects in the Jumbler rule are

  • character swapping, where you can configure the distance between the swapped positions
  • character insertion, where you can configure the number of character to insert and the alphabet of character you can insert
  • character change where you can configure the alphabet of changeable characters, the alphabet of characters you can insert

Obviously, you can mix several transformation at a time for the same source value.

The VERY valuable feature is that you can also produce a table with the original data, the corrupted data and the type of corruption applied to make sure that you detect all dirty data, only dirty data and if you miss ones then what were the type of corruption you missed.

Cool no ?

If you want to check that mechanism on a simple use case, you can register an account on GEDIS Studio at http://www.data-generator.com (its FREE) and import the "Testing" project from the shared workspace. Then go to the Production tab and launch generation of the file named "String Mutations.txt" it is based on the "String Mutations" generator.

In the following picture you can see the result of this sample applied over the OFAC list.

enter image description here

Do not hesitate to contact me if you have trouble with this.

Hope this helps

2

I started an article back in Dec (haven't finished) about handling bad data in government datasets. Perhaps there is enough info to seed bad data into an existing dataset at a predetermined distribution:

Dec. 18, 2013 By Andrew Ferlitsch

Problem: How we deal with bad data in government datasets.

While working with datasets from various government agencies, both US and abroad, as well as international organizations, we encounter what we call bad data. Bad data effects the value of the dataset and may cause problems in parsing the data, importation into databases, and indexing and search. Bad data generally falls into the following categories:

Wrong character encoding of dataset - results in garbled characters.
Misalignment of data - results in breaking non-adaptable parsing methods.
Errors in metadata descriptions - results in misinterpretation of fields.
Non-standard or esoteric representation of primary keys - results in problematic indexing of records.
Inconsistencies in delimination and use of punctuation with fields - results in problematic decomposition of the data.
Obsolete or historic data not identified accordingly - results in invalid (erroneous) records in database.

http://www.opengeocode.org/articles/baddata.php

1

I'm a big fan of twitter data because with the 140 characters of text per tweet comes a network and enourmous amounts of meta-data, including many incorrect computer-assigned tags (i.e. tweet language). What can you do with 140 characters and some meta-data: A recent example of researchers determining a user's location based on non-geotagged tweets (link).

Before you get started, you have to register as a developer and then you'll receive authentication keys (link). The interface is a ReST API - details.

I've previously posted two methods for collecting tweet data.

If you run the live stream, you can collect your own (you'll get more tweets than you can deal with).

Tweets from the live stream can come without any language filtering, and each tweet is assigned a language tweet[u'lang'] in addition to the user's language tweet[u'user'][u'lang']. You'll find that many of these auto-assignments are incorrect. For example, here is a graph of the languages of my tweets, which includes Slovene and Tagalog (link to tweet).

enter image description here

Another option is to search for tweets.

If you want to 'search' for all tweets, you can use lang:en as your search parameter. See here. Then you can loop over desired langauges.

In the tweets returned with lang:en you'll find many misspellings, abbreviations, etc.

The data comes in JSON format (details), which you can easily 'break' yourself by randomly replacing real characters with quotation marks. See here for an example (although start with 0.01%!

Not the answer you're looking for? Browse other questions tagged or ask your own question.