10

Are there any examples of open data that have been scraped from "real-world" sources? To explain what I mean by "real-world", here are some (made up) examples and then some criteria:

  • Train departure times from automated OCR of crowdsourced photos of train departure boards (UK readers will understand why this might be useful!)
  • Train departure information from mobile GPS measurements
  • Product nutrition information from OCR of crowdsourced photos of "nutrition information" on product labels (plus barcode scanning from image data)
  • As above, but product prices
  • Statistics about exercise habits from mobile GPS, accelerometer and temperature data
  • Statistics about noise pollution from mobile microphone measurements
  • Statistics about emotional state / mental health from processing photos of people on the web and analysing their facial expressions

These all involve some combination of the following criteria:

  1. Non-trivial processing of some sort of physical measurement source data: image processing / OCR, natural language processing, etc.
  2. The lack of an intrinsic need for direct user involvement in data gathering (other than consent, and app installation, if relevant). Currently with photo data, user involvement is likely required in order to take the photo and indicate that it be processed in this way, but this may change over time.
  3. Crowd-sourced source data gathering from mobile phone / tablet / google glass-type devices
  4. Source data (photos, text) fetched from the web
  5. Crowd-sourced data analysis (e.g. coding photographs of people by perceived emotional state based on facial expression)
  • does it have to be automated? CDDB, IMDB and plenty of other databases were done by people before 'crowd sourcing' was talked about. Or are you looking more for aspects of data repurposing (it was collected for one purpose, but then used for another) – Joe Jul 22 '13 at 21:01
  • I was trying to avoid defining it too precisely because I'm interested to hear about things that I hadn't considered. But no, I wasn't thinking about that kind of thing: IMDB doesn't really seem to meet very well any of my numbered criteria. Perhaps CDDB does, though. – Croad Langshan Jul 23 '13 at 21:52
3

The best example I have heard of is Real-time traffic monitoring using mobile phone data (PDF).

The idea is to derive road traffic velocity from the position data that the mobile phones within cars "generate" when moving from one base station to the next. The frequency of these base station handshakes approximates the travel velocity of the car. Practical application is the detection of congestion without the need for dedicated hardware.

  • Is there open data that uses this idea? – Croad Langshan Jul 23 '13 at 21:55
  • Probably not, as all mobile phone networks I know are operated by companies, and I see no reason for them to share monetizable, corporate data. – ojdo Aug 1 '13 at 12:51
2

Mobile (cell) phone data is more frequently being used for research purposes in a variety of fields and in very novel ways. Three examples to add to that provided by @ojdo include:

Geographic analysis of social divisions & interactions in France

Movement of malaria carriers in Kenya

Optimisation of bus routes in Ivory Coast

Regarding crowd-sourced photos, check out this fascinating demo of Photosynth from TED2007. One feature of Photosynth is to use images culled from the web (e.g., Flickr) to create large-scale montages (from about 3:40 in the video).

  • +1 for Photosynth demo (switch from fascinating to awesome beginning 3:00) – ojdo Jul 22 '13 at 13:34
  • Do all of these provide open data? – Croad Langshan Jul 23 '13 at 21:56
2

The Zooniverse team https://www.zooniverse.org/ has a bunch of great projects using crowd sourcing for research by reading in data or extracting measurements and category information from non-machine-readable records. Examples are extracting weather data from historic ships logs, digitized copies of botanical records, astronomical and biological images, etc.

1

I re-read the numbered list, and I don't know if it would qualify -- but there have been examples of people mining what people are talking about (twitter) and looking for (search queries) to extract information that you might not expect:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.