Questions tagged [nlp]

Natural language processing (NLP) is a subfield of artificial intelligence and machine learning that involves transforming or extracting useful information from natural language data.

Filter by
Sorted by
Tagged with
24
votes
6answers
28k views

Twitter open datasets

Where can I get Twitter datasets available for analysis? I found two: The May 2011 Calufa Twitter Scrape Cheng-Caverlee-Lee September 2009 - January 2010 Twitter Scrape
17
votes
4answers
10k views

Downloading the Panama Papers

I wonder where the Panama Papers can be downloaded. I have looked around, I couldn't find them. Where can I find them? What is the legality, and sensibility of requesting and downloading leaked ...
12
votes
4answers
6k views

Paraphrase data sets

I am looking for paraphrase data sets. I am aware of the following: PPDB: The Paraphrase Database (Ganitkevitch, Juri, Benjamin Van Durme, and Chris Callison-Burch. "PPDB: The Paraphrase Database." ...
11
votes
7answers
5k views

Large French dataset for NLP (not formal, rather discussions/reviews)

What am I looking for ? I'm looking for a big dataset (at least a few GB) containing French text. This french text should not have too much quality (such as literature or encyclopedias). Why this ...
10
votes
1answer
802 views

Where can I find a part-of-speech corpus free for commercial use?

I'm looking to teach myself more about NLP. I started with NLTK and I see that it has potential to eventually become something I could get paid for. From there, my journey continued to reading this ...
9
votes
5answers
335 views

Data set of news articles and scientific journals

I am working on a project in NLP area and in order to test the result I need to compare it against different data sources. I already have NYTimes data source. So, I am looking for another news source ...
8
votes
2answers
2k views

Medical Terminology in Patient Medical Records - Public Data Sets

I am interested in sample data of real patient medical records (anonymized or demographics removed completely) for the purpose of running through NLP system - specifically diagnoses, admissions and ...
7
votes
4answers
2k views

Tool to extract the main concepts/topics from web pages

I need to find the main themes or topics for each of 50,000 website urls. For example, http://austinzencenter.org/ should return topics such as zen, monastic, austin, retreat, buddha, etc. I'm ...
7
votes
4answers
248 views

Most common words in English

I am looking for a machine readable data set (e.g., a mere txt file with one word per row) containing the X most common words in English, excluding proper nouns (unlike this list). If possible, ...
7
votes
1answer
2k views

Spanish Sentiment Dataset

I'm looking for a Spanish language corpus tagged with sentiment (just positive & negative are needed, though neutral might be helpful.) I could use a Twitter dataset, but I'd prefer to use ...
7
votes
1answer
131 views

Corpus of human-scored machine translations?

Most of the parallel corpora (Opus, EuroParl, OpenSubtitles) have only human translations. src_txt | human_trans_of_src_txt_into_trg_lng (We assume all the translations are good.) Is there a corpus ...
7
votes
2answers
310 views

French equivalent of the brown corpus

The brown corpus is a collection of text where each element is already gramatically tagged. It contains about one million words and is often used to train statistical PoS taggers. For the french ...
6
votes
2answers
1k views

German news/text data set

I am looking for (possible buy) a dataset of German news data (i.e., the largest daily newspapers), where the data span at least ten years back up until now/recent. Any tips?
6
votes
2answers
97 views

Huge Biomedical Corpus for Unsupervised Experiments

I am looking for a very large collection ( > 10 GB) of biomedical text documents to run some unsupervised experiments with (for recognizing drug names, etc). Do we have access to such a thing for free?...
6
votes
1answer
217 views

British Newspaper Archive

Anyone know if there is a way to obtain the OCR txt files from the British Newspaper Archive via some bulk download API? Some more details: This is the British Newspaper Archive: http://www....
6
votes
1answer
277 views

Dataset for queries that are questions or others

I am looking for a dataset of queries that are posted in live chats or service providing chats. I want to test my API for tagging a query that is entered by the customer as a question or not a ...
5
votes
4answers
110 views

Tagged addresses

Does anyone know of a corpus of US addresses where the parts of the addresses have been tagged. Something like <streetNumber>123</streetNumber> <streetDirection>E.</...
5
votes
2answers
293 views

Do you know a technique for text mining judicial decisions?

I'm trying to find a method to mine sepecific informations in judicial decisions (judge's name, court, area of law and neutral citation number[canadian way to cite decisions uniformly across the ...
5
votes
1answer
3k views

ASCII Character Frequency Analysis

I've been looking for a table of the frequencies of all 128 ASCII characters, not just letters. The best table I've been able to find is this one, but that one only includes printable ASCII characters,...
5
votes
2answers
100 views

Downloadable word embeddings

I am looking for downloadable word embeddings (a.k.a. word vectors, distributed word representations). I'm aware of: word2vec GloVe SENNA as well as the retrofitting tool. What else is available?
5
votes
1answer
239 views

Actual text and corrected text? Or autocorrect data?

I seek essentially a parallel corpora, which has the dirty original user-generated input on the one side, and a cleaned up version on the other side - with corrected spelling, casing, punctuation and ...
5
votes
2answers
100 views

Are there translation probabilities for major language pairs?

Are there reasonable alignment probability files for major language pairs? Specifically, given a target language and a target word/phrase, what are the translation equivalents in other major ...
5
votes
2answers
404 views

Open text document corpus for information retrieval evaluation

INTRODUCTION: document collections (corpora) for evaluation of information retrieval (search engine) systems are pretty often behind a paywall. A notorious example is the TREC conference (http://trec....
5
votes
1answer
548 views

Dataset of Sentences Classified by Type?

Where can I find a labeled dataset of sentences classified by type (interrogative and declarative)? I'm particularly lookings for Russian sentences, but other open and labeled datasets of other ...
4
votes
2answers
47 views

Are there opendatasets of English terms describing nationalities, ethnicities?

Say one needs to parse text for any mention of nationality or ethnicity, or otherwise use these terms in the course of an NLP project. Is there a comprehensive resource of English words describing ...
4
votes
3answers
1k views

Datasets: Is there any public data-set for FAQ (Frequently asked questions) in different domains?

My thesis is about Analysis and Auto generation of FAQ in different domains. For conducting experiments, I need high volume of FAQs. That's the reason I am looking for a publicly available data-set ...
4
votes
2answers
87 views

Where can I find machine readable transcribed text of the 2016 Presidential speeches and debates?

I'm looking for a machine-readable repository of transcribed Presidential debates and speeches from the 2016 general election. Structured data with a common format is desirable, as well as updates for ...
4
votes
2answers
421 views

Semantic dictionary of English language

I am looking for a dictionary of English words tagged with their semantics. Like: red t:color green t:color long t:size t:max huge t:size t:max tiny t:size t:min hot ...
4
votes
3answers
97 views

Looking for conversation transcripts

I am looking for conversation transcripts. These can be from real sources such as radio or TV interviews, or even phone and IM captures, or fictional conversations such as plays and movies. I don't ...
4
votes
1answer
644 views

Datasets based on casual conversations for chatterbots

I am building a chatterbot that can answer questions related to tennis. I wish to know if there any any available datasets than can answer casual questions like "How are you", "How is the weather ...
4
votes
3answers
3k views

Dataset for english words of dictionary for a NLP project

I am working on a NLP project, and for that I need a dataset of English words (words typically found in dictionary). Could somebody please guide me how to put together such dataset? Is there any ...
4
votes
1answer
78 views

Speech recognition english corpuses

Could someone provide a list (or link to) of english speech recognition corpuses with their availability and size (preferably in hours).
4
votes
2answers
37 views

Non-ASCII Unicode symbols with their word equivalents

I am looking for a dataset containing Unicode symbols with some word equivalent. Examples: ©: copyright €: euro √: square root
4
votes
1answer
514 views

Where can I get calendar events dataset in text format?

I need to create a model in opennlp for identifying event information in text. I need a calendar events dataset for training this model. Can someone suggest any available open source dataset for ...
4
votes
1answer
52 views

English queries to GIS systems

I'm building a system that tries to answer geospatial English questions such as "Where is the closest restaurant" or "How many historic sites are in the east of Nantes?". Underlying database is OSM. ...
4
votes
1answer
302 views

Recommendation letter dataset

I am looking for a dataset containing academic recommendation letters, written in English, with as many following fields as possible: content of the letter date of the letter purpose of the letter (...
4
votes
1answer
53 views

License for data product of music lyrics?

I am interested in doing some natural language processing on copyrighted song lyrics. I saw a related project here, http://rappers.mdaniels.com.s3-website-us-east-1.amazonaws.com/ who used rap.genius....
4
votes
1answer
380 views

Is there any open dataset for named entity linking?

Entity linking is the task of detecting mentions of entities from a knowledge base in a document. Are there any openly available datasets to train and evaluate such systems? The datasets I came ...
4
votes
0answers
64 views

Corpus for intention mining

I need a corpus with intention words, but I was not able to get one yet. please somebody help me. Note : I'm new to intention mining if you know some useful corpus or links on intention mining ...
4
votes
0answers
58 views

Solved Programming Challenges Dataset

I'm looking for a dataset of solved programming challenges, like the ones on http://codeeval.com. The dataset should have the natural language description of the problem and N solutions for each ...
4
votes
0answers
263 views

Publicly available dataset of physician notes

I am looking for a publicly available data set of physician notes that describe medical reasoning, ideally freetext indexed by the level of training of the author. MIMIC-III has physiologic data, ...
3
votes
1answer
656 views

Physician notes with annotated PHI

I am looking for a data set of physician notes with annotated PHI (protected health information) as defined in Health Insurance Portability and Accountability Act (HIPAA)'s de-identification guide (US ...
3
votes
2answers
198 views

Is there any open data-set for sentence segmentation?

I am working on textual analysis of reports in English language. The documents are official records. I requires the sentence level segmentation of text. One way is through regular expression. Other ...
3
votes
3answers
3k views

Data sets for short text classification

I am looking for short text classifications (multilabel is OK). By short text I mean ~50 words max. The data set used in Xin Li, Dan Roth. Learning Question Classifiers. COLING. 2002. is a good ...
3
votes
2answers
879 views

SQL queries dataset

I am looking for a dataset containing natural language sentences (English preferred) and their corresponding SQL queries. E.g. "What are the names of the employees who live in Los Angeles" could be ...
3
votes
1answer
439 views

Looking for text-to-phoneme or pronunciation data for English words

I am looking for data of text and their corresponding phonemes for the purpose of training a classification tree and a neural network. Example: cat = k-a-t or something similar I am looking for at ...
3
votes
2answers
2k views

Offensive words - English and Spanish

Have a system generating four letter code and in testing just got up to AA-BJ ... and the BJ got me thinking. In a while the one after AR-SD could cause offence in the UK! Does anyone know of a list ...
3
votes
4answers
3k views

Where can I find a dataset containing legal documents?

I have a machine learning task I wish to pursue. For the task I will need several hundred sample legal documents of the following types: Employment contract, service contract, sale contract, rental ...
3
votes
1answer
534 views

Reddit comments labeled data-set for sentiment analysis?

I was searching for a Reddit comments data-set which is labeled into three classes: positive, negative and neutral to train a ML model. I can get labeled data for tweets, movie reviews but what about ...
3
votes
1answer
124 views

Dataset of documents and user libraries

I'm looking for a decently sized dataset (more than 100k docs) which consists of bag of words documents along with a list of users where associated to each user is a list of those documents the user ...