I'm developing a system to extract tags from text (English) and currently I have no dataset to test the system and evaluate, could someone point me to a source (preferably a free one) thanks. NOTE: By Tags I mean if there's an article about let's say new album of a musician then Tags should be something like ["music","new album","name of musician"] something along these lines.

  • 3
    Tags mapped to a document (e.g. newspaper articles) or to terms within these articles?
    – Suzana
    Mar 7, 2016 at 18:52

4 Answers 4


What you're trying to do is called named entity recognition. There exist several typical datasets for it, such as:

  • CoNLL-2002: free. 4 types of entities are tagged: locations, persons, organizations, and miscellaneous entities that do not belong in any of the three previous categories.

  • CoNLL-2003: free. 4 types of entities are tagged: locations, persons, organizations, and miscellaneous entities that do not belong in any of the three previous categories.

  • ACE: non-free.

  • OntoNotes Release 5.0 (free, release notes)

Entity types:

enter image description here

Excerpt from \ontonotes-release-5.0_LDC2013T19\ontonotes-release-5.0\data\files\data\english\annotations\bc\cnn\00\cnn_0006.name:

Live from <ENAMEX TYPE="ORG">CNN</ENAMEX> in <ENAMEX TYPE="GPE">Washington</ENAMEX> this is <ENAMEX TYPE="WORK_OF_ART">Late Edition with Wolf Blitzer</ENAMEX> /.
Good <ENAMEX TYPE="TIME">evening</ENAMEX> <ENAMEX TYPE="GPE">Washington</ENAMEX> /.

QTLEAP-2015-D5.11 - report on the state-of-the-art in NER and WSD gives a nice list of NER datasets:

enter image description here

enter image description here

  • Wikipedia NER: Ghaddar, Abbas, and Philippe Langlais. "WiNER: A Wikipedia annotated corpus for named entity recognition." In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 413-422. 2017.https://www.aclweb.org/anthology/I17-1042.pdf
  • Acronyms: Veyseh, Amir Pouran Ben, Franck Dernoncourt, Quan Hung Tran, and Thien Huu Nguyen. "What Does This Acronym Mean? Introducing a New Dataset for Acronym Identification and Disambiguation." In Proceedings of the 28th International Conference on Computational Linguistics, pp. 3285-3301. 2020. https://www.aclweb.org/anthology/2020.coling-main.292.pdf
  • Definitions: Spala, Sasha, Nicholas A. Miller, Yiming Yang, Franck Dernoncourt, and Carl Dockhorn. "DEFT: A corpus for definition extraction in free-and semi-structured text." In The 13th Linguistic Annotation Workshop, p. 124. 2019. https://www.aclweb.org/anthology/W19-4015.pdf
  • Physician notes with annotated PHI

A similar question was asked on stack overflow: Free Tagged Corpus for Named Entity Recognition (mirror)

  • In your answer you mention that OntoNotes is free. Where can one get it?
    – blambert
    Jan 13, 2018 at 1:30
  • @blambert from catalog.ldc.upenn.edu/LDC2013T19 (need to log in) Jan 13, 2018 at 1:44
  • Thanks. I poked around that page for 30min while logged in and couldn't find a link! Am I missing something?
    – blambert
    Jan 13, 2018 at 2:44
  • @blambert Did you log in? If so, you should see "Request Data" at the end of the page. Jan 13, 2018 at 2:57
  • I am logged in... but I don't see "Request Data". I'll get in touch with LDC.
    – blambert
    Jan 13, 2018 at 3:11

You may also want to check the American National Corpus and well as the holdings of the Linguistic Data Consortium.


Get the CoNLL-2003: data from this repo

The CoNLL-2003 shared task data files contain four columns separated by a single space. Each word has been put on a separate line and there is an empty line after each sentence. The first item on each line is a word, the second a part-of-speech (POS) tag, the third a syntactic chunk tag and the fourth the named entity tag. The chunk tags and the named entity tags have the format I-TYPE which means that the word is inside a phrase of type TYPE. Only if two phrases of the same type immediately follow each other, the first word of the second phrase will have tag B-TYPE to show that it starts a new phrase. A word with tag O is not part of a phrase.

This data set has 4 entity tags(Location, Person, Organisation and Misc) and it is used to train models for Named Entity Recognition tasks. The task is similar to the one proposed here just the entity tags are different.


You may consider SpaCY for pos_tagging. I bet you will love it spacy name entity recognition. It has got Name, Org, locations etc. It is faster in comparision to coreNLP, accuracy for NER holds at ~85%.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.