I'm currently searching for labeled datasets to use to train a model to extract named entities from informal text (think something similar to tweets). Because capitalization and grammar are often lacking in the documents in my dataset, I'm looking for out of domain data that's a bit more "informal" than the news articles and journal entries that many of today's state of the art named entity recognition systems are trained on. Any recommendations? So far I've only been able to locate 50k tokens from twitter published here: https://github.com/aritter/twitter_nlp/blob/master/data/annotated/ner.txt
-
what sort of named entities? people? places? something else?– JoeJul 2, 2014 at 13:06
-
People / places / music / movies / books, etc.– Madison MayJul 2, 2014 at 14:29
-
Are you open to using APIs for this instead of public domain lists? One that comes to mind is OpenCalais but there are many others as well– Mark SilverbergJul 2, 2014 at 17:44
-
Hi Mark -- I'm essentially trying to recreate the OpenCalais system for informal text. OpenCalais seems to perform well on well structured text, but typically performs poorly on informal text.– Madison MayJul 2, 2014 at 18:30
-
look into scraping microformats.– albertAug 5, 2014 at 19:38
Add a comment
|
2 Answers
Although the entity set is more restricted than you are looking for, the following might be useful:
https://github.com/sandeepAshwini/TwitterMovieData
The data is referenced in the following paper by Ashwini and Choi, which discusses and evaluates the general approach: http://arxiv.org/abs/1408.0782
The Social Security administration provides datasets on the top 1000 popular (baby) names per year. This would be a good starting resource for recognizing first names in feeds.
http://www.ssa.gov/OACT/babynames/index.html#ht=1
I would also consider adding in recognizing city names. There are a lot of free datasets. Below is a link to one of my own compilations built from the USGS and US NGA/GNS datasets.
answered Jul 1, 2014 at 18:11
-
I'm actually already using a gazetteer generated from the titles of all wikipedia pages. This means that I'm picking up all city names, first names, etc with no problem. The issues come when I need to pull out a named entity that isn't in my dataset (say because of a mispelling). Because of that, I'm trying to train a machine learning model to automatically extract named entities based on the context which they appear in. In other words, I'm looking for informal text with named entities of all types marked. See the twitter data linked for an example of what I'm searching for. Jul 1, 2014 at 19:17