14
votes
Downloading the Panama Papers
No leak
The Panama Papers were not leaked at all. They were handed over to a German media company which probably paid a good price (translation/summary here) for that deal.
This means, the panama ...
- 290
10
votes
Accepted
Downloading the Panama Papers
The Panama Papers database has just been published at this link
https://www.occrp.org/en/panamapapers/database.html
This ICIJ database contains information on almost 320,000 offshore entities that ...
- 332
8
votes
Paraphrase data sets
You definitely have the big ones with MSRP and PPDB.
A few others, grouped by maintainers.
CLiC at the Universitat de Barcelona
Maintain several highly annotated corpora.
These are available from ...
7
votes
Physician notes with annotated PHI
1) i2b2 2006 Deidentification and Smoking Challenge's data set:
NLP Data Set #1B: 889 de-identified discharge summaries with
de-identification challenge annotations, training and test sets and
...
- 7,740
6
votes
Paraphrase data sets
Entailment corpus: http://nlp.stanford.edu/projects/snli/
SemEval tasks: http://alt.qcri.org/semeval2015/index.php?id=tasks
SICK dataset: http://clic.cimec.unitn.it/composes/sick.html
- 161
6
votes
German news/text data set
I think a good source would be the COSMAS II-Korpora from the Institut für Deutsche Sprache.
It seems there is free registration and downloads for non-commercial use.
- 17.6k
6
votes
Most common words in English
It's not entirely open but the best I've found in this vein is www.wordfrequency.info - I am pretty sure that the 5000 word list is free, with fees for larger lists.
- 5,488
5
votes
Twitter open datasets
Check out these links below, and you can also search "twitter" in data.world for other related data. There's quite a few users who are already aggregating this data and creating analyses and cool ...
- 221
5
votes
ASCII Character Frequency Analysis
You can pretty easily create character (or word) frequency data based on a few lines of code. Here is python 2.7 code that collects counts of each character
# -*- coding: utf-8 -*-
import string ...
- 17.6k
4
votes
Paraphrase data sets
Google recently released paraphrase dataset: Paraphrase Adversaries from Word Scrambling (PAWS).
https://github.com/google-research-datasets/paws
This dataset is superior to many paraphrase datasets....
- 141
4
votes
Accepted
Medical Terminology in Patient Medical Records - Public Data Sets
The MIMIC-III dataset would be well suited to the kind of natural language processing (NLP) study that you are interested in doing. MIMIC-III includes deidentified nursing progress notes, imaging ...
- 1,446
4
votes
Datasets based on casual conversations for chatterbots
tl;dr - find and use AIML files
There is an open-source chat-box project called A.L.I.C.E.
Link to Demo
A.L.I.C.E. uses the AIML format
AIML (Artificial Intelligence Markup Language) is an XML-...
- 17.6k
4
votes
Dataset of common typos in English
Here is a link to several corpora of commonly misspelled words in several varieties of English - list includes the Birbeck Spelling Error corpus
4
votes
Accepted
Looking for text-to-phoneme or pronunciation data for English words
#1 - For phonological equivalents of English words check out the Irvine Phonotactic Online Dictionary (IPhOD) which is a collection of English words and non-words primarily intended for work in ...
4
votes
Semantic dictionary of English language
Eurovoc is a large semantic "Multilingual Thesaurus of the European Union"
- 141
4
votes
Accepted
Where can I find a dataset containing legal documents?
CFPB Credit Card Agreements DB I think that is a service contract.
docracy - open source legal contracts Requires sign up
- 11.9k
3
votes
Medical Terminology in Patient Medical Records - Public Data Sets
Another good-size data set of natural language patient notes is available here: http://www.mtsamples.com/
It covers notes across multiple specialities and conditions.
We can also provide an ...
- 39
3
votes
Downloadable word embeddings
Polyglot's webpage has word embeddings for ~ 137 languages, including English.
Distributed word representations (word embeddings) have recently contributed to competitive performance in language ...
- 7,740
3
votes
Looking for conversation transcripts
Open.edu is another good source for conversation transcripts of interviews, although you'll have ascertain them as I do not see a download button:
http://www.open.edu/openlearn/history-the-arts/...
- 11.9k
3
votes
Twitter open datasets
Here is a dataset of relative word frequency for the top 10,000 words in 890 million Tweets, divided by county: https://sites.google.com/site/wordmapperinfo/
- 351
3
votes
Datasets: Is there any public data-set for FAQ (Frequently asked questions) in different domains?
What about StackOverflow's own dataset? https://archive.org/details/stackexchange
- 131
3
votes
Paraphrase data sets
You can also check out the Quora Question Pairs Dataset which was used in the recent Kaggle competetition.
- 173
3
votes
Data sets for short text classification
The following repository contains 20000 questions from StackOverflow. https://github.com/jacoxu/StackOverflow . The data is provided by Kaggle. There are true labels as well. So you can evaluate the ...
- 31
3
votes
Tool to extract the main concepts/topics from web pages
Thanks Tasos for your very helpful DBpedia Spotlight Python code sample above. However, converting it from Python 2 to Python 3+ wasn't entirely straightforward so thought I'd post the revised version....
- 31
3
votes
German news/text data set
Event registry provides access to news articles from more than 100.000 news sources in 15 languages including German. The only problem is that you can only get access to the last 3 years of articles.
...
- 221
3
votes
ASCII Character Frequency Analysis
I wrote some code to analyse the reuters21578 corpus (thanks @philshem for pointing me to it)!
You can find the results here - ascii_freq.txt or ascii_freq.json. The code used to generate them can be ...
- 131
3
votes
Where can I get calendar events dataset in text format?
on OpenDataSoft you can find ~12 datasets with calendar events in them. You can export each of them in text format (CSV) or other format like Excel, ICS etc. You can also use the api if you prefer but ...
- 1,157
3
votes
Most common words in English
There are word frequency lists for a number of languages, including English, that were created by Hermit Dave from the data at opensubtitles.org.
You can download them for free on his blog. The ...
- 406
3
votes
Downloading the Panama Papers
It seems the media is not releasing it to the public yet.
There are some links floating around to a similar (but smaller) 2013 leak, but that is not the one that was released today.
http://www....
- 139
3
votes
Downloading the Panama Papers
Still waiting for the entire cache, but The Sunday Times has released a list with 37,000 names (10 April 2016).
Search our database of 37,000 names linked to Mossack Fonseca companies in the tax ...
- 17.6k
Only top scored, non community-wiki answers of a minimum length are eligible