Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [language]

For questions related to human language. Use 'programming' instead for questions related to programming languages.

21
votes
2answers
231 views

Data for word hierarchies

I'm looking for a dataset which will give me is-a and has-a type relations between senses (and give senses from words). WordNet seems to be the most popular model, but I'm trying to evaluate other ...
16
votes
2answers
3k views

IPA phonology database

Is there any open database of English words and their pronunciation in IPA? What about other languages (Arabic, French,...)? (Please also point out the license of the DB) Update: I tried to convert ...
13
votes
6answers
10k views

Speech audio files dataset with language labels

I am working on building a language classifier in speech/audio samples. I have been trying to find a dataset which may have considerable number of speech samples in various languages. The audio files ...
13
votes
1answer
2k views

Speech audio databases with phonemes labelled

Can anyone point me to opensource databases of speech audio with phonemes labelled, .wav or the like + label files, e.g. # time, phoneme 0.31200 ao 0.41200 th 0.54200 er ... I'd prefer English, ...
12
votes
7answers
1k views

List of English words

I'm writing a program that requires a list of English words, such as that used by a spell checker. I have been unable to find any such word lists whatsoever, even after trying 2 search engines.
12
votes
1answer
6k views

Large list of quotes

I am looking for a large database / list of quotes I can use. Preferably famous quotes / quotes by famous people. I have tried looking for a brainyquote API or something, but couldn't find one. If ...
12
votes
3answers
3k views

Is a spoken digit dataset available?

Is a labeled dataset of spoken digits, that means of people saying "zero", "one", "two", three", "four", "five", "six", "seven", "eight", or "nine" available? I would also be interested in such a ...
11
votes
2answers
2k views

American English SMS Text Message Corpora

I'm seeking corpora of American English SMS (<=160 characters) text messages. What corpora are available?
11
votes
6answers
3k views

Is there a free list of English word phonetics?

The list actually doesn't have to be free! I just need something that I can buy once or whatever. And if possible, but not needed, parts of speech. Before you say anything, I have googled for this ...
11
votes
1answer
3k views

Database of English words difficulty

Data request I am looking for a database that describes the "difficulty" of each English word. Difficulty of a word is not easy to define, but it is probably a mix of: Level of study needed to ...
10
votes
3answers
4k views

Database of English words pronunciation

Is there any downloadable open database of English words pronunciation in audio format?
10
votes
1answer
758 views

Where can I find a part-of-speech corpus free for commercial use?

I'm looking to teach myself more about NLP. I started with NLTK and I see that it has potential to eventually become something I could get paid for. From there, my journey continued to reading this ...
9
votes
1answer
1k views

Database with English words with grammatical classification?

Is there open database with English words with grammatical classification? I mean the list of words, with the information, which part of speech this word is (f.e. verb, noun, pronoun etc.).
9
votes
5answers
330 views

Data set of news articles and scientific journals

I am working on a project in NLP area and in order to test the result I need to compare it against different data sources. I already have NYTimes data source. So, I am looking for another news source ...
9
votes
3answers
4k views

Dataset of sentences translated into many languages

I'm looking for a dataset of human translated sentences. The ideal dataset would look like this: 1, en, The weather is nice today. 1, de, Das Wetter ist heute schön. 1, es, El clima es agradable hoy....
9
votes
2answers
751 views

List of abbreviations and acronyms

I am searching an list of abbreviations,acronyms which should be downloadable as a sql table or json or sth. like that so no api cause it might be not fast enough or has a limit like this one: http://...
9
votes
1answer
615 views

Text message corpus for American english

I am doing a research project to identify some patterns in how people interact with each other using text messages. The scope of the project is confined to American users. Can someone suggest a good ...
8
votes
6answers
3k views

Dataset of languages and where they are spoken (sub-national)

I am looking for a dataset describing (nearly) all languages, and where they are spoken. Bonus if the areas are linked to Geonames or another mainstream geographical dataset. Polygons are also OK. ...
8
votes
1answer
686 views

Phones dataset for speech recognition (not telephone number)

In phonetics a phone is a unit of speech sound. I am looking for a dataset of phone waveforms for speech recognition. There are about 17 vowels and 17 consonants for English phones. I tried looking ...
7
votes
1answer
131 views

Corpus of human-scored machine translations?

Most of the parallel corpora (Opus, EuroParl, OpenSubtitles) have only human translations. src_txt | human_trans_of_src_txt_into_trg_lng (We assume all the translations are good.) Is there a corpus ...
7
votes
2answers
295 views

French equivalent of the brown corpus

The brown corpus is a collection of text where each element is already gramatically tagged. It contains about one million words and is often used to train statistical PoS taggers. For the french ...
6
votes
4answers
4k views

Database world cities with their names in all languages of the world

I'm looking for a database of cities in the world with their names in all languages of the world. Where it is possible to download?
6
votes
2answers
965 views

German news/text data set

I am looking for (possible buy) a dataset of German news data (i.e., the largest daily newspapers), where the data span at least ten years back up until now/recent. Any tips?
6
votes
1answer
81 views

How to collect hand-writting data

I'm working on machine learning project that requires I collect hand-writing data of alphabets from an uncommon language (I'm 100% there is no available data out there). My question is how should I ...
6
votes
1answer
123 views

How can I get a list of synonyms that work across all contexts?

Many synonyms are context specific. For example "force" is a synonym for "drive" in the context of urging or inspiring, but not in the context of journeying by vehicle. Other synonyms seem to work ...
6
votes
1answer
183 views

Language Similarity Heuristic

I'm looking for a matrix of language pairs that shows some measure of their similarity. My main issue is that I don't know of heuristics that work well across different writing styles. Wikipedia's ...
6
votes
1answer
64 views

Phonemic & Syllabic N-Gram Distributions of the World's Languages

I am interested in doing some typological analyses by measuring the redundancy and IE entropy across languages in the phonemic and syllabic domains. I've searched for quite a bit but could not find ...
5
votes
1answer
421 views

Audio Datasets Featuring Different Speakers Saying the Same Sentence (English)?

Seeking a dataset of audio files consisting of recordings featuring different people saying the same sentence in English for deep neural network (DNN) training and testing.
5
votes
2answers
994 views

Word list for Common European Framework of Reference for Languages (CEFR)

The Common European Framework of Reference for Languages: Learning, Teaching, Assessment, abbreviated as CEFR, is a guideline used to describe achievements of learners of foreign languages across ...
5
votes
2answers
2k views

Frequency Analysis Character Distribution Data

Is there a location I can download character distributions for frequency analysis used in decryption attempt validation? I am specifically interested in ASCII value [32, 126] frequency distribution ...
5
votes
2answers
290 views

Do you know a technique for text mining judicial decisions?

I'm trying to find a method to mine sepecific informations in judicial decisions (judge's name, court, area of law and neutral citation number[canadian way to cite decisions uniformly across the ...
5
votes
1answer
195 views

Wikidata label language: How to fallback to ANY language?

In Wikidata, this shows labels in English, and falls back to Russian if no English is available: SERVICE wikibase:label { bd:serviceParam wikibase:language "en, ru" . } QUESTION: How to fall back ...
5
votes
2answers
471 views

Text data sources in German, primarily news articles

Can anyone recommend good sources for getting archived news texts in German language? Good source would satisfy the following conditions: recent in date (for example 2005-2015) variety of topics (...
5
votes
1answer
888 views

Longman English Dictionary Database

Is Longman English Dictionary as database available for public or should I digitalise it? I plan to create an app based on Longman English Dictionary, but I need the database somehow. Any idea how?
5
votes
2answers
99 views

Are there translation probabilities for major language pairs?

Are there reasonable alignment probability files for major language pairs? Specifically, given a target language and a target word/phrase, what are the translation equivalents in other major ...
5
votes
1answer
454 views

Dataset of Sentences Classified by Type?

Where can I find a labeled dataset of sentences classified by type (interrogative and declarative)? I'm particularly lookings for Russian sentences, but other open and labeled datasets of other ...
5
votes
1answer
120 views

Worldwide holidays, and their names in the local dominant language

Might be slightly OT here but I'll give it a try since the topic is probably well known to many here: We're looking for a database or system to get current and upcoming holidays by location, ...
5
votes
0answers
684 views

Where to download the Forvo database?

Forvo.com is a website where native speakers submit pronunciation of words. The audio data is under the BY-NC-SA license. I can't find any obvious place to download the whole data. There is an API, ...
4
votes
2answers
43 views

Are there opendatasets of English terms describing nationalities, ethnicities?

Say one needs to parse text for any mention of nationality or ethnicity, or otherwise use these terms in the course of an NLP project. Is there a comprehensive resource of English words describing ...
4
votes
2answers
68 views

Corpus of words divided by language

I am looking for a word dataset composed by lists of words of multiple languages (possibly all the major languages), non topic specific, divided by language. I don't really care for translations, ...
4
votes
2answers
87 views

Keyboard keystrokes with timing

Similar to this question: Where to find key log data for keyboard usage? ...but it doesn't have the answers I am looking for. I am researching keyboard layouts. I want to get timing data for each ...
4
votes
3answers
3k views

Dataset for english words of dictionary for a NLP project

I am working on a NLP project, and for that I need a dataset of English words (words typically found in dictionary). Could somebody please guide me how to put together such dataset? Is there any ...
4
votes
1answer
1k views

Dataset of English Poetry

Is there a dataset containing a group of poems in English? I couldn't find any resource online. I could web scrape poetry sites, but I want to avoid copyright issues and even a dataset of older poems ...
4
votes
1answer
2k views

ASCII Character Frequency Analysis

I've been looking for a table of the frequencies of all 128 ASCII characters, not just letters. The best table I've been able to find is this one, but that one only includes printable ASCII characters,...
4
votes
3answers
247 views

All Demonym's in their native language

COUNTRY - ? / ? US - American / Americans UK - English / English DE - Deutscher / Deutsche Anyone an idea where to find this? We're trying to find any sort of source for this for weeks now.
4
votes
1answer
41 views

I am looking for good german <-> czech dictionary in text format(or something else)

I am looking for good german <-> czech dictionary in text format (or something I can easy read). The Idea behind is I would like to program a program for myself for learning czech, but the sources ...
4
votes
2answers
61 views

Source of aggressively toned texts for machine learning

I was looking for a source where there would be an extended amount of texts with aggressive tone. It can be intertwined with non-aggressive texts, but it would be good if the ratio of aggressive texts ...
4
votes
1answer
35 views

Which licence and format should I use for the Norwegian language “data” I am creating?

While learning a foreign language, I am creating my own list of verbs (with their variants when conjugated), list of idioms, nouns (with plural form), etc. As an example, let's focus on the verbs: I ...
4
votes
2answers
435 views

Dataset for Named Entity Recognition on Informal Text

I'm currently searching for labeled datasets to use to train a model to extract named entities from informal text (think something similar to tweets). Because capitalization and grammar are often ...
4
votes
0answers
101 views

List of inverse and polarity flip words required

I'm a computer science student and I'm doing a research on sentimental analysis with a large database. Therefore, i need to create dictionaries to mapping words. Now i'm facing the problem of "...