While you will find some exceptions, sharing archived tweets is against Twitter Developer Policy.
Take all reasonable efforts to do the following, provided that when requested by Twitter, you must promptly take such actions:
Delete Content that Twitter reports as deleted or expired;
If a user deletes their own content, the archive should reflect that ...
The Panama Papers were not leaked at all. They were handed over to a German media company which probably paid a good price (translation/summary here) for that deal.
This means, the panama papers generate a lot of media attention, and probably already paid off for the Süddeutsche Zeitung but it wont have much impact as long as the full papers are ...
The Panama Papers database has just been published at this link
This ICIJ database contains information on almost 320,000 offshore entities that are part of the Panama Papers and the Offshore Leaks investigations. The data covers nearly 40 years up to the end of 2015 and links to people and companies in ...
I ended up creating my own system combining a bunch of APIs. Here is what I did:
I pushed this text out to various APIs to process and stored the results. I used alchemyapi.com, textrazor.com, opencalais.com. Those APIs have a lot of options but mainly I focused ...
You definitely have the big ones with MSRP and PPDB.
A few others, grouped by maintainers.
CLiC at the Universitat de Barcelona
Maintain several highly annotated corpora.
These are available from their site, by request/registration.
It is a request they grant with almost no questions asked.
(My experience was that after registering online I had to sent them ...
scraperwiki lets you search twitter for terms, but also allows twitters search operators. here's a quick search for my handle:
they also offer a get a list of followers feature, so not exactly what you are looking for, but a twitter tool nonetheless
EDIT: some sets i came across:
Twitter Data used for ...
You can use DBpedia Spotlight to extract semantic annotations from DBpedia. Here is the code for Python.
You will need these libraries:
The example is only for one link, but you can create a script to itterate through your url list.
from bs4 import BeautifulSoup
link = "http://...
1) i2b2 2006 Deidentification and Smoking Challenge's data set:
NLP Data Set #1B: 889 de-identified discharge summaries with
de-identification challenge annotations, training and test sets and
Please cite as:
Uzuner Ö., Juo Y, Szolovits P. "Evaluating the state-of-the-art in
automatic de-identification". J Am Med Inform ...
As far as I know, one of the best data sources for NLP has been the ENRON emails. While this is not a "news" source, it would certainly provide some raw data for you to play around with.
Another source is PubMed. While this doesn't provide you with the actual article information, it does provide you with a lot of abstracts for scientific writing. There have ...
I've been thinking about this question a lot and I have another solution. You correctly wrote that Wikipedia articles have too much quality due to the editing.
But, the discussion (talk) pages are full of raw text data and are not prone to being edited, or even correct.
An example for the French language Open Data page (source):
Y a-t-il une raison pour ...
WordNet is a (free) semantic database of the English language. You could query for example the complete hyponym tree of the term "inhabitant". Depending on your application, it could be a disadvantage that WordNet also contains historical ethnicities or nationalities.
For language names, an excellent database is Glottolog.
Once I compiled a dictionary of ...
Check out these links below, and you can also search "twitter" in data.world for other related data. There's quite a few users who are already aggregating this data and creating analyses and cool visualizations. If you're interested in collaborating, some are seeking additional contributors to help build out these datasets further. Hope this helps!
There are a several ways to get access to historical Twitter data. Here are some options:
I've had great luck using Topsy in looking at a wide variety of
tweets ranging from disease vectors in Africa to sentiment analysis.
Here's the link for 28K #opendata tweets for the last 20 days. You can
expand to "all time", search by language, and look at influencers....
If you are interested to just scrape some data from web pages, then like @magdmartin wrote, you just need to write some code to download the HTML (python requests package) and then to parse the HTML (BeautifulSoup in this case).
from bs4 import BeautifulSoup
r = requests.get('https://www.canlii.org/en/ca/scc/doc/2007/2007scc4/...
Not exactly SQL but SPARQL:
NL-SPARQL: A Dialog-System Challenge Set for Converting Natural Language to Structured Queries
NL-SPARQL is a data set of natural language (NL) utterances to a conversational system in the movies domain and corresponding queries to Freebase in SPARQL. This dataset was collected via Crowdsourcing as described below.
The data set ...
After doing some searching it seems that the Moby Project is in the public domain, and they have a POS corpus. However, it's simply a dictionary so it doesn't help with words that have multiple POS. Also it's not encoded in ASCII so opening it up in a text editor, it's hard to read. Will obviously require pre-processing before it can be useful.
I will ...
Here's a list of all the offensive words banned from Google's "What Do You Love?" site:
Wiktionary has a pretty thorough list of English Euphemisms:
Wiktionary also has a (seemingly) scarce list of Spanish Euphemisms:
Google recently released paraphrase dataset: Paraphrase Adversaries from Word Scrambling (PAWS).
This dataset is superior to many paraphrase datasets. As stated in their page,
Existing paraphrase identification datasets lack sentence pairs that have high lexical overlap without being paraphrases. Models ...
The MIMIC-III dataset would be well suited to the kind of natural language processing (NLP) study that you are interested in doing. MIMIC-III includes deidentified nursing progress notes, imaging reports, and discharge summaries for tens of thousands of patients who were admitted to intensive care units in the US.
You can read about several NLP studies that ...
You can pretty easily create character (or word) frequency data based on a few lines of code. Here is python 2.7 code that collects counts of each character
# -*- coding: utf-8 -*-
import string # definitions of ascii printable chars
from collections import defaultdict # fast counting
d = defaultdict(int) # define dictionary for counting ...
tl;dr - find and use AIML files
There is an open-source chat-box project called A.L.I.C.E.
Link to Demo
A.L.I.C.E. uses the AIML format
AIML (Artificial Intelligence Markup Language) is an XML-compliant language that's easy to learn, and makes it possible for you to begin customizing an Alicebot or creating one from scratch within minutes.
#1 - For phonological equivalents of English words check out the Irvine Phonotactic Online Dictionary (IPhOD) which is a collection of English words and non-words primarily intended for work in psycholinguistics and speech perception. You can download the entire database and see phonological entries like the following:
A corresponding table shows ...