Is there a freely available test dataset of research papers in computer science field? I need to evaluate my text retrieval system which uses a computer science domain ontology called CSO, so the dataset should come with a ground truth/relevance judgments for text retrieval task and/or classification task.


There are some freely available collections of research papers, but whether they are annotated in the way that you need is a different story. Unless the kind of task that you're targeting has been done before, it's likely you'll have to do some annotation work yourself.

I'm aware of the ACL Anthology Corpus, which is about CS but more specifically NLP papers.

Another option to investigate would be Medline and/or PMC data: while this is about biomedical research, some of the journals it indexes overlap with computer science. This it's so massive there might be enough CS data in there. The advantage of Medline is that the abstracts are indexed with Mesh descriptors which are often used as a kind of gold standard, although I don't know if it would contain enough CS concepts.

  • Actually, the problem with ACL corpus is that there is little intersection between its topics and CSO ontology topics. I think the same applies to Medline and PMC. I need a dataset whose papers are specifically in the domain of computer science, with relevance judgments – Nina Jan 24 at 17:09
  • 2
    @Nina you're a bit optimistic :) it's unlikely there's some available data which fits your needs perfectly, I'm afraid. Your best chance is probably to obtain the data used to create the CSO itself, or to have a sample manually annotated. – Erwan Jan 24 at 19:08
  • Why don't you add your last suggestion to the answer so that I accept it? – Nina Jan 26 at 20:42
  • @Nina thanks for your suggestion, it's not really an answer to the question so I prefer not to. You never know, maybe somebody will give you an answer with a better idea of data. Good luck with your data collection and experiments! – Erwan Jan 26 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.