5

INTRODUCTION: document collections (corpora) for evaluation of information retrieval (search engine) systems are pretty often behind a paywall. A notorious example is the TREC conference (http://trec.nist.gov/). Apart from money, they ask for affiliation, making the data unavailable for the hobbyist/open-source. A few older corpora exist (http://www.daviddlewis.com/resources/testcollections/reuters21578/) but these are "antique" in ways that make then less useful for the evaluation of current systems. Others like the Wikipedia dumps(1), the European Union multilingual corpora (2)(often used in automatic translation) or the RFCs (3) could be used but are not standard in the research community.

QUESTION: does any open text document corpus of a decent size and quality in English exist, that is widely used for evaluation of information retrieval systems? The widespread usage puts the result of one's own evaluation into a meaningful context.

5

2 Answers 2

1

You might look at How can I Share my Data Sets Without Worrying About Copyright Issues?

I don't know if the author is willing to share or not. He was concerned about copyright issues.

3
  • Well, a corpus of open-license text would be a first step. A lot of sources with a permissive license come into mind (Wikipedia, Project Gutenberg, Usenet etc.). The question is how to make that corpus community standard? Sep 22, 2015 at 7:30
  • The first step would be to define a community. Then decide who is going to be the arbiter of that community's standards.
    – pndfam05
    Sep 22, 2015 at 14:07
  • ..........SIGIR Sep 23, 2015 at 10:34
1

If you're interested in Twitter search, you can prep your own tweet IR collection using the open tools described on the TREC Microblog (MB) 2011/2012 pages. The test queries and judgements are also openly available from TREC, as is the tool for evaluating (trec_eval)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.