4

I'm looking for a proper data set for my anomaly detection model. At this point, we are looking for a parameter which has the following characteristic: it is a data stream that gradually (or even quickly) changes through the time.

We want to detect anomalies in such data stream with our own model. Could you please help me by introducing a parameter that has the above feature and also it is worth to detect outliers in that?

Improve this question
5
  • What about population census? The data have usually a smooth evolution, unless a war, a disaster, that can make the data change quickly
    – user_0
    Commented Jun 27, 2015 at 8:43
  • Would crime, traffic accidents, air quality, or other data work? I think so... I can provide some links to some with both bulk download as well as APIs if you want.
    – Mark Silverberg
    Commented Jun 27, 2015 at 16:29
  • Thanks @user_0, it is interesting I'll talk to my supervisor about it.
    – Boshra N
    Commented Jun 28, 2015 at 1:49
  • @Skram I think some of those might be very interesting. That would be awesome if you could provide some links for me as well!
    – Boshra N
    Commented Jun 28, 2015 at 1:52
  • Is there any parameter in internet network
    – Boshra N
    Commented Jun 28, 2015 at 1:55

3 Answers 3

Reset to default
5

The Global Population Dynamics Database has thousands of time series from biological populations recording changes in population size over time. Maybe that would be useful to you?

Prendergast J , Bazeley-White E , Smith O , Lawton J , Inchausti P , Kidd D , and Knight S. 2010. The Global Population Dynamics Database. The KNB Data Repository. (doi:10.5063/F1BZ63Z8)

There are also many other environmental time series such as temperature, wind speed, currents, and other abiotic parameters over time in the KNB Repository, some of which are relatively high frequency, such as this temperature time series, which is one of many:

Partnership for Interdisciplinary Studies of Coastal Oceans (PISCO) , Menge B , and Chan F. PISCO: Intertidal: site temperature data: Boiler Bay, Oregon, USA (BBYX00) (doi:10.6085/AA/BBYX00_XXXITV2XLSR02_20120112.50.1)

Improve this answer
1
  • Thanks a lot for your complete and detailed answer! Regarding population size I'm not sure how anomaly detection might be beneficial but still it is worth considering!
    – Boshra N
    Commented Jun 29, 2015 at 3:35
3

Any space weather data would likely fit the bill. Both NOAA and NASA have links to various data sources.

Most earth weather data would likely also fit what you're asking for (although I'm not as familiar with it). Some simple time series data would be temperature or USGS river gauges.

Improve this answer
2
  • Thanks a lot, I didn't know that those valid institutions have provided such data sets.
    – Boshra N
    Commented Jun 29, 2015 at 3:29
  • 1
    More specifically, weather.noaa.gov/pub/SL.us008001/DF.an/DC.sflnd/DS.metar (and similar) have lots of data (updated every few minutes) with plenty of anomalies, and it would be great if someone wrote a program to find and remove these!
    – user3856
    Commented Jul 5, 2015 at 17:18
2

I'm not sure the nature of the anomalous patterns you're seeking, so would fine-grain financial streams work for you? I'm assuming you're looking for free datasets. I've used several of the sources listed here: http://www.quantshare.com/sa-426-6-ways-to-download-free-intraday-and-tick-data-for-the-us-stock-market to develop pattern-break models for both short and long term transaction performance of individual stocks and markets. In particular, I find the Google Finance data to be easy to access and of course, plentiful.

Depending on the frequency of readings you need, there is everything from frequent intraday data (1 minute reporting) through daily data (24 hour interval) and historical data. If you want higher frequency than that, some of the sources do streams for all buy-sell data, but that is universally a paid feature. Good luck!

Improve this answer
1
  • 1
    Very interesting. I need to also consult with an economist to see how anomaly detection can help in this area.
    – Boshra N
    Commented Jun 29, 2015 at 3:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.