Hot answers tagged

11

The largest English corpus I've found (over 10,000 messages) is the National University of Singapore's SMS corpus -- select the corpus with "all" messages -- however, closer examination reveals that relatively few of the messages originate from US participants. A corpus of SMS spam messages has been created which are written in English. There are over 1,000 ...


9

What you're trying to do is called named entity recognition. There exist several typical datasets for it, such as: CoNLL-2002: free. 4 types of entities are tagged: locations, persons, organizations, and miscellaneous entities that do not belong in any of the three previous categories. CoNLL-2003: free. 4 types of entities are tagged: locations, persons, ...


5

I've been thinking about this question a lot and I have another solution. You correctly wrote that Wikipedia articles have too much quality due to the editing. But, the discussion (talk) pages are full of raw text data and are not prone to being edited, or even correct. An example for the French language Open Data page (source): Y a-t-il une raison pour ...


5

You may be able to use the International Picture Naming Project (IPNP), but I think the total pictures are way less than 10k. Here's a link to the query tool. You may get lucky searching through "image recognition" research websites. Here's one compilation of online resources, and a particular one.


5

Probably the Swedish Academian Wordlist (SAOL) is the most authoritative source in this field. There exists SAOL for Android, the .obb expansion file is a regular SQLite database. This list was extracted from the .obb database file. The list contains more than 90 000 nouns (whereas the list from dict.cc contains less than 9 000 nouns). About 75% of nouns ...


4

Not exactly SQL but SPARQL: NL-SPARQL: A Dialog-System Challenge Set for Converting Natural Language to Structured Queries NL-SPARQL is a data set of natural language (NL) utterances to a conversational system in the movies domain and corresponding queries to Freebase in SPARQL. This dataset was collected via Crowdsourcing as described below. The ...


4

I don't know a corpus, but I know a way to create one. If you know how to program you can use the Facebook API and download all the public facebook status from USA with their comments. Then you can use them as a corpus. Info about Facebook API


4

I created a dataset with of on-line data. It has 369 symbols (including a-z A-Z 0-9 \alpha-\omega), but it is online data. You will have to create the rendered versions yourself: http://write-math.com/data Each class has at least 50 recordings. (I don't have much data for letters, so it will probably not be more.) edit: I've redered it. See The HASYv2 ...


4

If you are just interested in words (and not translations) you can check out Wortschatz at Universität Leipzig. An alternative (with very different coverage across languages, but actually providing translations) is the Wiktionary.


3

Arxiv.org is a repository of scientific pre-prints. License API Bulk data access Many papers can be accessed in TeX typesetting format (in addition to PDF, PS).


3

Some ideas coming to my mind: Wikibooks Wikipedia (dump) other specialised wikis, e.g. for cooking recipes fan fiction sites obvious, but you may overlook it: The content of the stack exchange sites (Science Fiction and World Building may be of special interest to you)


3

We release large corpora of German news for the annual machine translation shared task organised by the WMT conference. Unfortunately it is shuffled (so you cannot reconstruct the original articles) for the reasons given above. The latest release is here: http://www.statmt.org/wmt16/translation-task.html For 2014, we did actually release a crawl from the ...


3

Blurbs, or short descriptive material to promote a book, may be impossible to legally share do to individual copyrights of the authors or publishers. The Goodreads API has many endpoints, and they include this note: Book cover images, descriptions, and other data from third party sources might be excluded, because we do not have a license to distribute ...


3

You can get portfolios that by law are submitted to the US Securities and Exchange Committee (SEC) Form 13F—Reports Filed by Institutional Investment Managers An institutional investment manager that uses the U.S. mail (or other means or instrumentality of interstate commerce) in the course of its business, and exercises investment discretion over $...


3

This page lists some on/off-line handwriting database for academic use. Some of them can be downloaded free while others may need application. http://www.gavo.t.u-tokyo.ac.jp/~qiao/database.html


3

I am not 100% sure if the following links are useful for you. Please let me know if you are looking for something else. http://www.iitg.ac.in/ece/emstlab/SRdatabase/introduction.php http://accent.gmu.edu/howto.php http://www.signalprocessingsociety.org/technical-committees/list/sl-tc/spl-nl/2012-05/the-rss2015-speech-corpus/


3

I recently made a corpus available of text messages from Spanish-dominant bilingual young adults in New York City (Bilingual Youth Texts Corpus). It is at www.byts.commons.gc.cuny.edu


3

How about movies? Option 1: bulk download of French language movie subtitles. I haven't tried it, but OpenSubtitles.org has both instructions for programming and non-programming. Option 2: Download plot synopses from the plaintext IMDB database (with reviews) - mirror links. Useful files are 'language.list', 'biographies.list', 'plot.list', etc. ...


3

I cannot help you with products review and you have already got an answer for Twitter. But if you need a dataset with plain text in french, the best solution is the Wikipedia Dump. I used it for another project in English for a similar reason (sentiment analysis and information extraction). Here is the download link with all the info. The small dataset is 2....


3

I can answer for the Twitter data. You can gather plenty of French language tweets by one of two methods: Live Stream. In this case, you can pass this URL: https://stream.twitter.com/1.1/statuses/sample.json?language=fr and you will receive tweets as the standard JSON dictionary. An example python code to download a portion of the public stream is here. In ...


3

http://openaddresses.io/ It's an initiative to collect open (CC0) addresses around Unites States and globally. You could transform these as you like..


3

Most telephone recording datasets are privately owned by parties who have access to such data like for example call-centers and tech-support companies etc. This data is subject to various privacy laws (which differ from one country to another), therefore it is tricky to find such data. Nevertheless, the CallHome database and the CallFriend database -by ...


3

Project Gutenberg offers 57.000 free books, available in different formats. Among them, utf-8 encoded plain text with minimal formatting. The NLTK comes with access to a range of corpora. Among them, a selection from Project Gutenberg, and a chat corpus (if you are looking for more colloquial use of English). Beware of the varying licenses that apply. If ...


2

frWaC: frWaC: a 1.6 billion word corpus constructed from the Web limiting the crawl to the .fr domain and using medium-frequency words from the Le Monde Diplomatique corpus and basic French vocabulary lists as seeds. The corpus was POS-tagged and lemmatized with the TreeTagger, more information available here. Reference: Baroni, Marco, et al. "The WaCky ...


2

You can collect email data in .mbox text files using a search engine. For example, a google search for filetype:mbox from results in plenty of .mbox data. My best attempt at French-only .mbox files is this search. Individual .mbox files can be concatenated into one large file (linux method): cat mboxfile1 > mboxfile echo >> mboxfile cat mboxfile2 &...


2

You can use Texas A&M's Address Parsing and Normalization service to turn any set of addresses into a tagged set of addresses.


2

If the number of addresses is thousands at a time, and not more, you can use the Google Geocoding API for free. You can even use the API without a key for small sets of addresses. The URL looks like this: http://maps.googleapis.com/maps/api/geocode/json?sensor=false&address=Museum für Gestaltung Zurich Switzerland And returns a JSON with structured ...


2

microformats do that, off hand look @ web data commons for a large repo: http://webdatacommons.org/ more real world examples from microformats wiki: http://microformats.org/wiki/dataset-examples#Real-World_Examples


2

Based on @NeilSlater's comment, you can easily calculate character N-grams with a few lines of code. In this snippet, I use Python's Collections library, which is quite fast for these types of applications: from collections import defaultdict def make_char_ngram(text,N): data = defaultdict(int) # for speed for i in xrange(len(text)-N+1): ...


2

I actually ended up paying folks on MechanicalTurk to label questions as important/unimportant from a couple of news articles I downloaded. There are 410 sentences total, which I have on my github here.


Only top voted, non community-wiki answers of a minimum length are eligible