I'm looking for a proper data set for my anomaly detection model. At this point, we are looking for a parameter which has the following characteristic: it is a data stream that gradually (or even quickly) changes through the time.

We want to detect anomalies in such data stream with our own model. Could you please help me by introducing a parameter that has the above feature and also it is worth to detect outliers in that?

  • What about population census? The data have usually a smooth evolution, unless a war, a disaster, that can make the data change quickly – user_0 Jun 27 '15 at 8:43
  • Would crime, traffic accidents, air quality, or other data work? I think so... I can provide some links to some with both bulk download as well as APIs if you want. – Mark Silverberg Jun 27 '15 at 16:29
  • Thanks @user_0, it is interesting I'll talk to my supervisor about it. – Boshra N Jun 28 '15 at 1:49
  • @Skram I think some of those might be very interesting. That would be awesome if you could provide some links for me as well! – Boshra N Jun 28 '15 at 1:52
  • Is there any parameter in internet network – Boshra N Jun 28 '15 at 1:55

The Global Population Dynamics Database has thousands of time series from biological populations recording changes in population size over time. Maybe that would be useful to you?

Prendergast J , Bazeley-White E , Smith O , Lawton J , Inchausti P , Kidd D , and Knight S. 2010. The Global Population Dynamics Database. The KNB Data Repository. (doi:10.5063/F1BZ63Z8)

There are also many other environmental time series such as temperature, wind speed, currents, and other abiotic parameters over time in the KNB Repository, some of which are relatively high frequency, such as this temperature time series, which is one of many:

Partnership for Interdisciplinary Studies of Coastal Oceans (PISCO) , Menge B , and Chan F. PISCO: Intertidal: site temperature data: Boiler Bay, Oregon, USA (BBYX00) (doi:10.6085/AA/BBYX00_XXXITV2XLSR02_20120112.50.1)

  • Thanks a lot for your complete and detailed answer! Regarding population size I'm not sure how anomaly detection might be beneficial but still it is worth considering! – Boshra N Jun 29 '15 at 3:35

Any space weather data would likely fit the bill. Both NOAA and NASA have links to various data sources.

Most earth weather data would likely also fit what you're asking for (although I'm not as familiar with it). Some simple time series data would be temperature or USGS river gauges.

  • Thanks a lot, I didn't know that those valid institutions have provided such data sets. – Boshra N Jun 29 '15 at 3:29
  • 1
    More specifically, weather.noaa.gov/pub/SL.us008001/DF.an/DC.sflnd/DS.metar (and similar) have lots of data (updated every few minutes) with plenty of anomalies, and it would be great if someone wrote a program to find and remove these! – user3856 Jul 5 '15 at 17:18

I'm not sure the nature of the anomalous patterns you're seeking, so would fine-grain financial streams work for you? I'm assuming you're looking for free datasets. I've used several of the sources listed here: http://www.quantshare.com/sa-426-6-ways-to-download-free-intraday-and-tick-data-for-the-us-stock-market to develop pattern-break models for both short and long term transaction performance of individual stocks and markets. In particular, I find the Google Finance data to be easy to access and of course, plentiful.

Depending on the frequency of readings you need, there is everything from frequent intraday data (1 minute reporting) through daily data (24 hour interval) and historical data. If you want higher frequency than that, some of the sources do streams for all buy-sell data, but that is universally a paid feature. Good luck!

  • 1
    Very interesting. I need to also consult with an economist to see how anomaly detection can help in this area. – Boshra N Jun 29 '15 at 3:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.