10

I would like to demonstrate outlier / anomaly detection and for that I need a real-life dataset.

I found the post

http://www.researchgate.net/post/What_are_anomaly_detection_benchmark_datasets

helpful, but I search for something simple and obvious for teaching purposes. I want to run a Kernel Density Estimation and assume that a data point which is at a position with low density (according to KDE) is an outlier.

Does someone of you know of such a dataset?

2

4 Answers 4

3

Try looking at the data here ! https://datamarket.com/data/list/?q=

Looots of time series. I've found some nice anomaly sets in there. Specfically, time series.


Are you a student ? If so, you can request Yahoo's outlier dataset !


Have you checked out the datasets at quandl? They aren't made for outlier detection but you can definitely find anomalies if you just spend a little time looking.


You might also appreciate this project: Numenta Anomaly Benchmark (NAB)

This is taken directly from the readme in NAB/tree/master/data.

NAB Data Corpus

Data are ordered, timestamped, single-valued metrics. All data files contain anomalies, unless otherwise noted.

Real data

  • realAWSCloudwatch/

    AWS server metrics as collected by the AmazonCloudwatch service. Example metrics include CPU Utilization, Network Bytes In, and Disk Read Bytes.

  • realAdExchange/

    Online advertisement clicking rates, where the metrics are cost-per-click (CPC) and cost per thousand impressions (CPM). One of the files is normal, without anomalies.

  • realKnownCause/

    This is data for which we know the anomaly causes; no hand labeling.

    • ambient_temperature_system_failure.csv: The ambient temperature in an office setting.
    • cpu_utilization_asg_misconfiguration.csv: From Amazon Web Services (AWS) monitoring CPU usage – i.e. average CPU usage across a given cluster. When usage is high, AWS spins up a new machine, and uses fewer machines when usage is low.
    • ec2_request_latency_system_failure.csv: CPU usage data from a server in Amazon's East Coast datacenter. The dataset ends with complete system failure resulting from a documented failure of AWS API servers. There's an interesting story behind this data in the Numenta blog
    • machine_temperature_system_failure.csv: Temperature sensor data of an internal component of a large, industrial mahcine. The first anomaly is a planned shutdown of the machine. The second anomaly is difficult to detect and directly led to the third anomaly, a catastrophic failure of the machine.
    • nyc_taxi.csv: Number of NYC taxi passengers, where the five anomalies occur during the NYC marathon, Thanksgiving, Christmas, New Years day, and a snow storm. The raw data is from the NYC Taxi and Limousine Commission. The data file included here consists of aggregating the total number of taxi passengers into 30 minute buckets.
    • rogue_agent_key_hold.csv: Timing the key holds for several users of a computer, where the anomalies represent a change in the user.
    • rogue_agent_key_updown.csv: Timing the key strokes for several users of a computer, where the anomalies represent a change in the user.
  • realRogueAgent/

    This data represents computer usage patterns for different users, where an anomaly may occur with a rogue user of the computer.

  • realTraffic/

    Real time traffic data from the Twin Cities Metro area in Minnesota, collected by the Minnesota Department of Transportation. Included metrics include occupancy, speed, and travel time from specific sensors.

  • realTweets/

    A collection of Twitter mentions of large publicly-traded companies such as Google and IBM. The metric value represents the number of mentions for a given ticker symbol every 5 minutes.

Artificial data

  • artificialNoAnomaly/

    Artifically-generated data without any anomalies.

  • artificialWithAnomaly/

    Artifically-generated data with varying types of anomalies.

3

Perhaps use something like the Hertzsprung-Russell Diagram Data of Star Cluster CYG OB1, it's a well-known dataset with outliers.

The data set is used in:
Robust Regression and Outlier Detection
P. J. Rousseeuw and A. M. Leroy (1987) ; Wiley, p.27, table 3.

2

We published a large-scale study on the performance of k-nearest-neighbor based outlier detection.

On the Evaluation of Unsupervised Outlier Detection:
Measures, Datasets, and an Empirical Study

by G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, I. Assent and M. E. Houle
Data Mining and Knowledge Discovery, 2016, DOI: 10.1007/s10618-015-0444-8

The supplementary material which includes over 20 data sets, in multiple variants, as well as precomputed results for many algorithms and a reproducibility package is online at:

1

River View from Numenta stream data from public data source while keeping their historic values.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.