The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Hot answers tagged

14

While you will find some exceptions, sharing archived tweets is against Twitter Developer Policy. Take all reasonable efforts to do the following, provided that when requested by Twitter, you must promptly take such actions: Delete Content that Twitter reports as deleted or expired; etc If a user deletes their own content, the archive ...


14

No leak The Panama Papers were not leaked at all. They were handed over to a German media company which probably paid a good price (translation/summary here) for that deal. This means, the panama papers generate a lot of media attention, and probably already paid off for the Süddeutsche Zeitung but it wont have much impact as long as the full papers are ...


10

The Panama Papers database has just been published at this link https://www.occrp.org/en/panamapapers/database.html This ICIJ database contains information on almost 320,000 offshore entities that are part of the Panama Papers and the Offshore Leaks investigations. The data covers nearly 40 years up to the end of 2015 and links to people and companies in ...


8

You definitely have the big ones with MSRP and PPDB. A few others, grouped by maintainers. CLiC at the Universitat de Barcelona Maintain several highly annotated corpora. These are available from their site, by request/registration. It is a request they grant with almost no questions asked. (My experience was that after registering online I had to sent ...


7

I ended up creating my own system combining a bunch of APIs. Here is what I did: I pulled the homepage text from each website and cleaned up all the html, javascript and styles I pushed this text out to various APIs to process and stored the results. I used alchemyapi.com, textrazor.com, opencalais.com. Those APIs have a lot of options but mainly I focused ...


7

72 hours #gamergate scrape provides a link to download the dataset.


7

scraperwiki lets you search twitter for terms, but also allows twitters search operators. here's a quick search for my handle: https://scraperwiki.com/dataset/crt52ai/view/fpzct4y they also offer a get a list of followers feature, so not exactly what you are looking for, but a twitter tool nonetheless EDIT: some sets i came across: Twitter Data used for ...


7

1) i2b2 2006 Deidentification and Smoking Challenge's data set: NLP Data Set #1B: 889 de-identified discharge summaries with de-identification challenge annotations, training and test sets and ground truth. Please cite as: Uzuner Ö., Juo Y, Szolovits P. "Evaluating the state-of-the-art in automatic de-identification". J Am Med Inform ...


6

You can use DBpedia Spotlight to extract semantic annotations from DBpedia. Here is the code for Python. You will need these libraries: BeautifulSoup urllib2 urllib json The example is only for one link, but you can create a script to itterate through your url list. from bs4 import BeautifulSoup import urllib2 import json import urllib link = "http://...


6

Entailment corpus: http://nlp.stanford.edu/projects/snli/ SemEval tasks: http://alt.qcri.org/semeval2015/index.php?id=tasks SICK dataset: http://clic.cimec.unitn.it/composes/sick.html


6

It's not entirely open but the best I've found in this vein is www.wordfrequency.info - I am pretty sure that the 5000 word list is free, with fees for larger lists.


5

As far as I know, one of the best data sources for NLP has been the ENRON emails. While this is not a "news" source, it would certainly provide some raw data for you to play around with. Another source is PubMed. While this doesn't provide you with the actual article information, it does provide you with a lot of abstracts for scientific writing. There have ...


5

I've been thinking about this question a lot and I have another solution. You correctly wrote that Wikipedia articles have too much quality due to the editing. But, the discussion (talk) pages are full of raw text data and are not prone to being edited, or even correct. An example for the French language Open Data page (source): Y a-t-il une raison pour ...


5

WordNet is a (free) semantic database of the English language. You could query for example the complete hyponym tree of the term "inhabitant". Depending on your application, it could be a disadvantage that WordNet also contains historical ethnicities or nationalities. For language names, an excellent database is Glottolog. Once I compiled a dictionary of ...


5

I think a good source would be the COSMAS II-Korpora from the Institut für Deutsche Sprache. It seems there is free registration and downloads for non-commercial use.


5

#1 - For phonological equivalents of English words check out the Irvine Phonotactic Online Dictionary (IPhOD) which is a collection of English words and non-words primarily intended for work in psycholinguistics and speech perception. You can download the entire database and see phonological entries like the following: A corresponding table shows ...


4

Check out these links below, and you can also search "twitter" in data.world for other related data. There's quite a few users who are already aggregating this data and creating analyses and cool visualizations. If you're interested in collaborating, some are seeking additional contributors to help build out these datasets further. Hope this helps! https://...


4

There are a several ways to get access to historical Twitter data. Here are some options: I've had great luck using Topsy in looking at a wide variety of tweets ranging from disease vectors in Africa to sentiment analysis. Here's the link for 28K #opendata tweets for the last 20 days. You can expand to "all time", search by language, and look at influencers....


4

What you're trying to do seems to fall under "topic modeling" or "clustering." Looking at the DiRT wiki's list of clustering tools, perhaps this topic-modeling tool would be a good place to start.


4

If you are interested to just scrape some data from web pages, then like @magdmartin wrote, you just need to write some code to download the HTML (python requests package) and then to parse the HTML (BeautifulSoup in this case). import requests from bs4 import BeautifulSoup import re r = requests.get('https://www.canlii.org/en/ca/scc/doc/2007/2007scc4/...


4

Not exactly SQL but SPARQL: NL-SPARQL: A Dialog-System Challenge Set for Converting Natural Language to Structured Queries NL-SPARQL is a data set of natural language (NL) utterances to a conversational system in the movies domain and corresponding queries to Freebase in SPARQL. This dataset was collected via Crowdsourcing as described below. The ...


4

After doing some searching it seems that the Moby Project is in the public domain, and they have a POS corpus. However, it's simply a dictionary so it doesn't help with words that have multiple POS. Also it's not encoded in ASCII so opening it up in a text editor, it's hard to read. Will obviously require pre-processing before it can be useful. I will ...


4

What about StackOverflow's own dataset? https://archive.org/details/stackexchange


4

Here's a list of all the offensive words banned from Google's "What Do You Love?" site: https://gist.github.com/jamiew/1112488 Wiktionary has a pretty thorough list of English Euphemisms: https://en.wiktionary.org/wiki/Category:English_euphemisms Wiktionary also has a (seemingly) scarce list of Spanish Euphemisms: https://en.wiktionary.org/wiki/...


4

The MIMIC-III dataset would be well suited to the kind of natural language processing (NLP) study that you are interested in doing. MIMIC-III includes deidentified nursing progress notes, imaging reports, and discharge summaries for tens of thousands of patients who were admitted to intensive care units in the US. You can read about several NLP studies that ...


4

tl;dr - find and use AIML files There is an open-source chat-box project called A.L.I.C.E. Link to Demo A.L.I.C.E. uses the AIML format AIML (Artificial Intelligence Markup Language) is an XML-compliant language that's easy to learn, and makes it possible for you to begin customizing an Alicebot or creating one from scratch within minutes. The ...


4

It seems the media is not releasing it to the public yet. There are some links floating around to a similar (but smaller) 2013 leak, but that is not the one that was released today. http://www.thereportertimes.com/panama-papers-icij-offshore-leaks-database-documents/23489/ https://archive.org/download/OffshoreLeaksDatabase This appears to have the high-...


4

Here is a link to several corpora of commonly misspelled words in several varieties of English - list includes the Birbeck Spelling Error corpus


4

Eurovoc is a large semantic "Multilingual Thesaurus of the European Union"


Only top voted, non community-wiki answers of a minimum length are eligible