Hot answers tagged

14 votes

Downloading the Panama Papers

No leak The Panama Papers were not leaked at all. They were handed over to a German media company which probably paid a good price (translation/summary here) for that deal. This means, the panama ...
  • 290
10 votes
Accepted

Downloading the Panama Papers

The Panama Papers database has just been published at this link https://www.occrp.org/en/panamapapers/database.html This ICIJ database contains information on almost 320,000 offshore entities that ...
  • 332
8 votes

Paraphrase data sets

You definitely have the big ones with MSRP and PPDB. A few others, grouped by maintainers. CLiC at the Universitat de Barcelona Maintain several highly annotated corpora. These are available from ...
7 votes

Twitter open datasets

72 hours #gamergate scrape provides a link to download the dataset.
  • 181
7 votes

Physician notes with annotated PHI

1) i2b2 2006 Deidentification and Smoking Challenge's data set: NLP Data Set #1B: 889 de-identified discharge summaries with de-identification challenge annotations, training and test sets and ...
6 votes

Paraphrase data sets

Entailment corpus: http://nlp.stanford.edu/projects/snli/ SemEval tasks: http://alt.qcri.org/semeval2015/index.php?id=tasks SICK dataset: http://clic.cimec.unitn.it/composes/sick.html
6 votes

Most common words in English

It's not entirely open but the best I've found in this vein is www.wordfrequency.info - I am pretty sure that the 5000 word list is free, with fees for larger lists.
  • 5,488
5 votes

Where can I find a part-of-speech corpus free for commercial use?

After doing some searching it seems that the Moby Project is in the public domain, and they have a POS corpus. However, it's simply a dictionary so it doesn't help with words that have multiple POS. ...
5 votes

Are there opendatasets of English terms describing nationalities, ethnicities?

WordNet is a (free) semantic database of the English language. You could query for example the complete hyponym tree of the term "inhabitant". Depending on your application, it could be a disadvantage ...
  • 406
5 votes

Data set of news articles and scientific journals

As far as I know, one of the best data sources for NLP has been the ENRON emails. While this is not a "news" source, it would certainly provide some raw data for you to play around with. Another ...
5 votes

Twitter open datasets

Check out these links below, and you can also search "twitter" in data.world for other related data. There's quite a few users who are already aggregating this data and creating analyses and cool ...
5 votes

German news/text data set

I think a good source would be the COSMAS II-Korpora from the Institut für Deutsche Sprache. It seems there is free registration and downloads for non-commercial use.
  • 17.4k
5 votes

ASCII Character Frequency Analysis

You can pretty easily create character (or word) frequency data based on a few lines of code. Here is python 2.7 code that collects counts of each character # -*- coding: utf-8 -*- import string ...
  • 17.4k
4 votes

SQL queries dataset

Not exactly SQL but SPARQL: NL-SPARQL: A Dialog-System Challenge Set for Converting Natural Language to Structured Queries NL-SPARQL is a data set of natural language (NL) utterances to a ...
4 votes

Twitter open datasets

There are a several ways to get access to historical Twitter data. Here are some options: I've had great luck using Topsy in looking at a wide variety of tweets ranging from disease vectors in Africa ...
  • 4,497
4 votes
Accepted

Offensive words - English and Spanish

Here's a list of all the offensive words banned from Google's "What Do You Love?" site: https://gist.github.com/jamiew/1112488 Wiktionary has a pretty thorough list of English Euphemisms: https://...
  • 11.8k
4 votes

Paraphrase data sets

Google recently released paraphrase dataset: Paraphrase Adversaries from Word Scrambling (PAWS). https://github.com/google-research-datasets/paws This dataset is superior to many paraphrase datasets....
4 votes
Accepted

Medical Terminology in Patient Medical Records - Public Data Sets

The MIMIC-III dataset would be well suited to the kind of natural language processing (NLP) study that you are interested in doing. MIMIC-III includes deidentified nursing progress notes, imaging ...
  • 1,436
4 votes

Datasets based on casual conversations for chatterbots

tl;dr - find and use AIML files There is an open-source chat-box project called A.L.I.C.E. Link to Demo A.L.I.C.E. uses the AIML format AIML (Artificial Intelligence Markup Language) is an XML-...
  • 17.4k
4 votes

Dataset of common typos in English

Here is a link to several corpora of commonly misspelled words in several varieties of English - list includes the Birbeck Spelling Error corpus
4 votes
Accepted

Looking for text-to-phoneme or pronunciation data for English words

#1 - For phonological equivalents of English words check out the Irvine Phonotactic Online Dictionary (IPhOD) which is a collection of English words and non-words primarily intended for work in ...
4 votes

Semantic dictionary of English language

Eurovoc is a large semantic "Multilingual Thesaurus of the European Union"
4 votes
Accepted

Where can I find a dataset containing legal documents?

CFPB Credit Card Agreements DB I think that is a service contract. docracy - open source legal contracts Requires sign up
  • 11.8k
3 votes

Datasets: Is there any public data-set for FAQ (Frequently asked questions) in different domains?

What about StackOverflow's own dataset? https://archive.org/details/stackexchange
3 votes

Datasets: Is there any public data-set for FAQ (Frequently asked questions) in different domains?

You can download FAQs from github, although they won't really be cross-domain... Github Advanced Search Example of a simple query that returns some FAQs files (will require some fine-tuning, of ...
  • 17.4k
3 votes

Data sets for short text classification

The following repository contains 20000 questions from StackOverflow. https://github.com/jacoxu/StackOverflow . The data is provided by Kaggle. There are true labels as well. So you can evaluate the ...
3 votes

Tool to extract the main concepts/topics from web pages

Thanks Tasos for your very helpful DBpedia Spotlight Python code sample above. However, converting it from Python 2 to Python 3+ wasn't entirely straightforward so thought I'd post the revised version....
  • 31
3 votes

Twitter open datasets

Here is a dataset of relative word frequency for the top 10,000 words in 890 million Tweets, divided by county: https://sites.google.com/site/wordmapperinfo/
  • 351

Only top scored, non community-wiki answers of a minimum length are eligible