I'm currently searching for labeled datasets to use to train a model to extract named entities from informal text (think something similar to tweets). Because capitalization and grammar are often lacking in the documents in my dataset, I'm looking for out of domain data that's a bit more "informal" than the news articles and journal entries that many of today's state of the art named entity recognition systems are trained on. Any recommendations? So far I've only been able to locate 50k tokens from twitter published here: https://github.com/aritter/twitter_nlp/blob/master/data/annotated/ner.txt

  • what sort of named entities? people? places? something else? – Joe Jul 2 '14 at 13:06
  • People / places / music / movies / books, etc. – Madison May Jul 2 '14 at 14:29
  • Are you open to using APIs for this instead of public domain lists? One that comes to mind is OpenCalais but there are many others as well – Mark Silverberg Jul 2 '14 at 17:44
  • Hi Mark -- I'm essentially trying to recreate the OpenCalais system for informal text. OpenCalais seems to perform well on well structured text, but typically performs poorly on informal text. – Madison May Jul 2 '14 at 18:30
  • look into scraping microformats. – albert Aug 5 '14 at 19:38

Although the entity set is more restricted than you are looking for, the following might be useful:


The data is referenced in the following paper by Ashwini and Choi, which discusses and evaluates the general approach: http://arxiv.org/abs/1408.0782

| improve this answer | |

The Social Security administration provides datasets on the top 1000 popular (baby) names per year. This would be a good starting resource for recognizing first names in feeds.


I would also consider adding in recognizing city names. There are a lot of free datasets. Below is a link to one of my own compilations built from the USGS and US NGA/GNS datasets.


| improve this answer | |
  • I'm actually already using a gazetteer generated from the titles of all wikipedia pages. This means that I'm picking up all city names, first names, etc with no problem. The issues come when I need to pull out a named entity that isn't in my dataset (say because of a mispelling). Because of that, I'm trying to train a machine learning model to automatically extract named entities based on the context which they appear in. In other words, I'm looking for informal text with named entities of all types marked. See the twitter data linked for an example of what I'm searching for. – Madison May Jul 1 '14 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.