18

ConceptNet is a semantic network containing lots of things computers should know about the world, especially when understanding text written by people. Trying to reproduce your relation-sequences yields: finger PartOf hand IsA body part, which looks surprisingly "dead end". chair IsA seat RelatedTo furniture MadeOf wood ...


14

Might want to double check the license, but the baseline standard is the CMU Pronunciation dictionary, which is freely downloadable and also ships with many NLP libraries, like NLTK (python). For out-of-vocabulary words, I've had great success with Sequitur G2P, which is both trainable and under the GPL. edit: note that CMUDict (and many other speech ...


14

There are plenty of open corpora (databases) of English words available. Specifically, take a look at Brown and WordNet. Check out Natural Language Toolkit it is written in Python and has those corpora available for download. It is one of the most popular packages to work with human languages data. If you prefer to use web based API, take a look at Wordnik ...


14

Have you seen WikiQuote by the Wikimedia Foundation? There is an API endpoint at http://en.wikiquote.org/w/api.php which uses the standard MediaWiki API for there are API clients in many different languages. EDIT: Two WikiQuote API-specific links: https://stackoverflow.com/questions/13762688/wiki-quotes-api and http://bwgz57.wordpress.com/2013/02/14/in-...


13

You can download an Aspell Dictionary, then convert it to simple list of words: aspell -d en dump master | aspell -l en expand > my.dict A few other dictionaries.


11

English EuroWordNet at globalwordnet.org is worth considering, depending on your requirements. Opencyc.org and Google freebase.com have large taxonomies already available for download.


11

The largest English corpus I've found (over 10,000 messages) is the National University of Singapore's SMS corpus -- select the corpus with "all" messages -- however, closer examination reveals that relatively few of the messages originate from US participants. A corpus of SMS spam messages has been created which are written in English. There are over 1,000 ...


9

Use wordnet, it's an academic project but a de facto for various purposes including the one you asked for. There are also higher semantics modeled that you can use for various functions. It's also convenient to query and integrate it into your app.


9

You can use the Tatoeba website which has full sentences in text and audio as downloads. Sentences with audio Download http://downloads.tatoeba.org/exports/sentences_with_audio.tar.bz2 Fields and structure Sentence id File description Contains the ids of the sentences, in all languages, for which audio is available. Thanks to ...


8

Wikidata has a category called "city" which will give you a list of world cities https://www.wikidata.org/wiki/Q515 You can use SPARQL to query Wikidata https://www.mediawiki.org/wiki/Wikidata_query_service/User_Manual https://query.wikidata.org/ A sample SPARQL query will give you the city name in the target language SELECT ?city ?cityLabel ?country ?...


7

If you're on a Mac, there's a list of words, 1 per line, in the file /usr/share/dict/words. The usr folder is hidden by default, but you can view it by following the following steps: Open the Finder. In the menu, go to Go->Go to Folder or press Cmd-Shift-G. In the box, type /usr/share/dict. You'll see a list of files. Copy the file called words to whatever ...


7

The MRC Psycholinguistic Database is similar to what you're looking for. It is somewhat out of date in terms of design, but it does allow lookup of English words, pronunciation, and a number of other psychologically interesting facts about words.


7

Wikimedia Commons currently offers more than 20.000 sound files with English pronunciation, around 1.500 of those with British English pronunciation. All of them are published under an open license. Unfortunately, there are currently no dumps of the media files available. However, there is a page that explains how to reuse the content outside of Wikimedia. ...


7

https://github.com/Jakobovski/free-spoken-digit-dataset is a free spoken digit dataset (FSDD). As an added bonus it comes with a few useful python utility functions. I created this dataset because I had the same problem. Please contribute to increase the dataset's size.


7

I suggest looking at the SpokenLanguages2 dataset, which is very extensive, containing a bit more than one hour of speech for each of the 172 languages: https://community.topcoder.com/longcontest/?module=ViewProblemStatement&rd=16555&pm=13978 The audio is very high quality. Although they do not provide access to the testing set, you have to do your ...


7

I guess the Geonames project is the closest answer to your question. You can download the data directly on the website or you can also find it here and download it/use the api. Cheers


7

The speech accent archive: uniformly presents a large set of speech samples from a variety of language backgrounds. As for English, the following subset contains the sentences: Please call Stella. Ask her to bring these things with her from the store: Six spoons of fresh snow peas, five thick slabs of blue cheese, and maybe a snack for her ...


6

The New General Service List (updated just a few weeks ago) is a list of "the most important words for second language learners of English". It ranks the top three thousand headwords (plus more words within each headword, such as plurals and tenses). So it won't cover all the words in your list, and it's really indicating "usefullness" rather than ...


6

The CIA Word Factbook has a field listing for languages spoken per country and percentage of population that speaks it. https://www.cia.gov/library/publications/the-world-factbook/fields/2098.html For finer grain, you can Google 'language spoken home dataset'. This will find census datasets related to the distribution of languages spoken. Below are some ...


6

Tatoeba.org is exactly that: http://tatoeba.org Their data is human-edited and used by dozens of products/websites including electronic dictionaries, so it is of reasonable quality. It has 471468 English sentences and 179 languages. (not all sentences are translated to all languages though, far from that) The structure is not a simple 1-1-1, a sentence ...


6

UCLA Phonetics Lab Archive offers many audio files of spoken word in many languages, although you'll have to dig through and mix-match to your liking: http://archive.phonetics.ucla.edu/ IPA Handbook downloads actually seems to be exactly what you are asking for, however its only for personal use; any other usage requires contacting them for permission...so ...


6

Maybe you can do something with https://rhinospike.com/language/ It's a bit like Tatoeba mentioned above edit : Take also a look at librivox (public domain audiobooks). It comes with an API, then you could send a request like https://librivox.org/api/feed/audiobooks?fields={url_zip_file,language}&offset=XXX&limit=YYY to extract files' id


6

The first step is splitting the image into character arrays. To do that, check out the answers in this question: Separate image of text into component character images. In particular, the ImageMagick answer from 2015. (If you can determine how the input is given, then collect the characters as separate images.) To convert the image into a 2D array, you can ...


6

Probably the Swedish Academian Wordlist (SAOL) is the most authoritative source in this field. There exists SAOL for Android, the .obb expansion file is a regular SQLite database. This list was extracted from the .obb database file. The list contains more than 90 000 nouns (whereas the list from dict.cc contains less than 9 000 nouns). About 75% of nouns ...


5

If you're trying to use Google, the general terms to use would be : From the linguisitics community: 'text corpora' or 'text corpus' (http://en.wikipedia.org/wiki/List_of_text_corpora) or For the password cracking community : 'wordlist' (one word, might need to keep quoted; http://www.openwall.com/wordlists/)


5

Second try: Google Ngram Viewer contains raw counts of 1-, 2-, ...-grams of text, retrieved from its book scanning endeavor. The section 1-grams contains counts of the occurence of lettres, numbers and even punctuation. They are provided as tab-separated value files, so the frequencies should be derivable with modest scripting efforts. Found via Wikipedia ...


5

As far as I know, one of the best data sources for NLP has been the ENRON emails. While this is not a "news" source, it would certainly provide some raw data for you to play around with. Another source is PubMed. While this doesn't provide you with the actual article information, it does provide you with a lot of abstracts for scientific writing. There have ...


5

WordNet is a (free) semantic database of the English language. You could query for example the complete hyponym tree of the term "inhabitant". Depending on your application, it could be a disadvantage that WordNet also contains historical ethnicities or nationalities. For language names, an excellent database is Glottolog. Once I compiled a dictionary of ...


5

I'm not aware of an available dataset in the requested form, but it should be possible to obtain something similar by parsing data from wiktionary. The latest dump can be obtained at https://dumps.wikimedia.org/enwiktionary/latest/ Also, some interesting links can be found by googling "wiktionary rdf"


Only top voted, non community-wiki answers of a minimum length are eligible