INTRODUCTION: document collections (corpora) for evaluation of information retrieval (search engine) systems are pretty often behind a paywall. A notorious example is the TREC conference (http://trec.nist.gov/). Apart from money, they ask for affiliation, making the data unavailable for the hobbyist/open-source. A few older corpora exist (http://www.daviddlewis.com/resources/testcollections/reuters21578/) but these are "antique" in ways that make then less useful for the evaluation of current systems. Others like the Wikipedia dumps(1), the European Union multilingual corpora (2)(often used in automatic translation) or the RFCs (3) could be used but are not standard in the research community.

QUESTION: does any open text document corpus of a decent size and quality in English exist, that is widely used for evaluation of information retrieval systems? The widespread usage puts the result of one's own evaluation into a meaningful context.


You might look at How can I Share my Data Sets Without Worrying About Copyright Issues?

I don't know if the author is willing to share or not. He was concerned about copyright issues.

  • Well, a corpus of open-license text would be a first step. A lot of sources with a permissive license come into mind (Wikipedia, Project Gutenberg, Usenet etc.). The question is how to make that corpus community standard? – Jason Doeson Sep 22 '15 at 7:30
  • The first step would be to define a community. Then decide who is going to be the arbiter of that community's standards. – pndfam05 Sep 22 '15 at 14:07
  • ..........SIGIR – Jason Doeson Sep 23 '15 at 10:34

If you're interested in Twitter search, you can prep your own tweet IR collection using the open tools described on the TREC Microblog (MB) 2011/2012 pages. The test queries and judgements are also openly available from TREC, as is the tool for evaluating (trec_eval)

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.