I'm looking for datasets in which any quantity is estimated by an individual person. These are datasets in the tradition of Galton (1907), in which individuals estimated the weight of an ox.

Ideally the individual persons would be making these estimates without consultation with other individuals or resources, but this is not essential.

I need to know the true value for the estimate.

Galton, F. (1907). Vox populi (the wisdom of crowds). Nature, 75, 450-451.


1 Answer 1


Online contests may provide a source of this data. For example, I don't understand this tweet, but there are hundreds of replies each with a guess around 900-1000.

enter image description here

There are many Twitter contests like this, and the good thing about Twitter is that you can download tweets with code (via the Twitter Public Search API) and then parse and analyse the results. For example, I could download all the responses to this tweet, and the easily parse out integer numbers.

See here for more details about the API.

In terms of formulating a good search string, I'd start with contest+guess and sort by Top (not Live). With the API you can only search back 1 week or so.

  • 1
    I have that "Mining the Social Web" book you linked and also some basic knowledge of how to use the Twitter API. However, according to stackoverflow.com/questions/2693553/… there is no direct way to download all responses to a particular Tweet. How would you download all the replies? Aug 23, 2015 at 5:34
  • 1
    With the GET search endpoint I would first search to:@user and then in the results I would select based on the "in_reply_to_status_id" field in the JSON response, by matching it to the particular tweet.
    – philshem
    Aug 23, 2015 at 9:15
  • 1
    You wrote: "I could download all the responses to this tweet, and the easily parse out integer numbers." Given that the tweet in question is more than 7 days ago, is it nonetheless possible to download the responses using the API? Aug 23, 2015 at 13:35
  • 1
    Not exactly with the Search function, so you'd have to find recent contests. You can download old tweets if you get the user timeline (up to 3600 tweets, or so) or if you know the tweet id. For this exact contest, you can scroll all the way to the bottom, hit Control+U (Firefox) to see the HTML source, and then parse the responses out of the HTML source. This can be done even manually (or grep if you have linux, mac).
    – philshem
    Aug 24, 2015 at 5:36
  • Curiously, the CTRL-U option doesn't work for this competition. When I scroll to the end and press CTRL-U it only includes data from the very first replies. For example, the final guess (1352 entrants) does not appear. Aug 24, 2015 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.