Questions tagged [language]

For questions related to human language. Use 'programming' instead for questions related to programming languages.

Filter by
Sorted by
Tagged with
7
votes
4answers
5k views

Database world cities with their names in all languages of the world

I'm looking for a database of cities in the world with their names in all languages of the world. Where it is possible to download?
6
votes
1answer
3k views

ASCII Character Frequency Analysis

I've been looking for a table of the frequencies of all 128 ASCII characters, not just letters. The best table I've been able to find is this one, but that one only includes printable ASCII characters,...
3
votes
2answers
95 views

Keyboard keystrokes with timing

Similar to this question: Where to find key log data for keyboard usage? ...but it doesn't have the answers I am looking for. I am researching keyboard layouts. I want to get timing data for each ...
6
votes
2answers
1k views

German news/text data set

I am looking for (possible buy) a dataset of German news data (i.e., the largest daily newspapers), where the data span at least ten years back up until now/recent. Any tips?
7
votes
1answer
134 views

Corpus of human-scored machine translations?

Most of the parallel corpora (Opus, EuroParl, OpenSubtitles) have only human translations. src_txt | human_trans_of_src_txt_into_trg_lng (We assume all the translations are good.) Is there a corpus ...
2
votes
0answers
60 views

American English pronunciation dictionary in text form showing syllables

Looking for pronunciation and syllabification data for American English. I want to analyze pronunciation and spelling rules/heuristics to see how accurate they are and identify exceptions. The goal ...
2
votes
2answers
31 views

Common words missing from typical spellcheckers

I find the spellcheckers included with most software to be woefully inadequate. Many common words are not included in your typical spellchecker. For example, try spellchecking these common words: ...
14
votes
6answers
11k views

Speech audio files dataset with language labels

I am working on building a language classifier in speech/audio samples. I have been trying to find a dataset which may have considerable number of speech samples in various languages. The audio files ...
4
votes
0answers
105 views

List of inverse and polarity flip words required

I'm a computer science student and I'm doing a research on sentimental analysis with a large database. Therefore, i need to create dictionaries to mapping words. Now i'm facing the problem of "...
6
votes
1answer
128 views

How can I get a list of synonyms that work across all contexts?

Many synonyms are context specific. For example "force" is a synonym for "drive" in the context of urging or inspiring, but not in the context of journeying by vehicle. Other synonyms seem to work ...
2
votes
3answers
113 views

Single-word Translations in All Languages

I am looking for a dataset of single-word translations, essentially a translation dictionary for all languages. I don't need all words, but simply a set of very common words: "yes", "no", "true", "...
4
votes
2answers
500 views

Text data sources in German, primarily news articles

Can anyone recommend good sources for getting archived news texts in German language? Good source would satisfy the following conditions: recent in date (for example 2005-2015) variety of topics (...
4
votes
3answers
302 views

All Demonym's in their native language

COUNTRY - ? / ? US - American / Americans UK - English / English DE - Deutscher / Deutsche Anyone an idea where to find this? We're trying to find any sort of source for this for weeks now.
4
votes
3answers
3k views

Dataset for english words of dictionary for a NLP project

I am working on a NLP project, and for that I need a dataset of English words (words typically found in dictionary). Could somebody please guide me how to put together such dataset? Is there any ...
4
votes
1answer
125 views

Worldwide holidays, and their names in the local dominant language

Might be slightly OT here but I'll give it a try since the topic is probably well known to many here: We're looking for a database or system to get current and upcoming holidays by location, ...
4
votes
2answers
62 views

Source of aggressively toned texts for machine learning

I was looking for a source where there would be an extended amount of texts with aggressive tone. It can be intertwined with non-aggressive texts, but it would be good if the ratio of aggressive texts ...
3
votes
2answers
2k views

Offensive words - English and Spanish

Have a system generating four letter code and in testing just got up to AA-BJ ... and the BJ got me thinking. In a while the one after AR-SD could cause offence in the UK! Does anyone know of a list ...
2
votes
1answer
73 views

Mailing list datasets

Related to this question: Obtaining personal mail corpus I'm looking for a list of archived public mailing lists from diverse communities and in diverse languages. For example, a list would look ...
3
votes
3answers
149 views

For natives of each language, most common second language and proportion of people fluent in it

I am trying to prioritize what languages my software should be translated to. Regardless of other factors, I want to first look at how many people are native speakers of each language, example: ...
10
votes
1answer
826 views

Where can I find a part-of-speech corpus free for commercial use?

I'm looking to teach myself more about NLP. I started with NLTK and I see that it has potential to eventually become something I could get paid for. From there, my journey continued to reading this ...
4
votes
2answers
48 views

Are there opendatasets of English terms describing nationalities, ethnicities?

Say one needs to parse text for any mention of nationality or ethnicity, or otherwise use these terms in the course of an NLP project. Is there a comprehensive resource of English words describing ...
1
vote
0answers
27 views

Tourism/survival sentences in most languages

I am looking for the first 50 or 100 sentences/words that an English-only-speaking tourist would learn when staying in a non-English-speaking country for a few weeks. For instance, the first few ...
2
votes
1answer
212 views

Context-aware dataset of movies or music

Looking for access to context aware dataset(s) of movies or music. Context aware dataset should contain user related information, such as user gender, mood, time, social, location, weather, emotions ...
6
votes
1answer
69 views

Phonemic & Syllabic N-Gram Distributions of the World's Languages

I am interested in doing some typological analyses by measuring the redundancy and IE entropy across languages in the phonemic and syllabic domains. I've searched for quite a bit but could not find ...
3
votes
1answer
219 views

source of character-level ngrams that include spaces

I'm looking for a source of character-level ngrams and counts that include spaces. I've found character-level ngram datasets, but they don't include spaces. Ideally, the dataset will include up to 8-...
11
votes
2answers
1k views

List of abbreviations and acronyms

I am searching an list of abbreviations,acronyms which should be downloadable as a sql table or json or sth. like that so no api cause it might be not fast enough or has a limit like this one: http://...
9
votes
5answers
349 views

Data set of news articles and scientific journals

I am working on a project in NLP area and in order to test the result I need to compare it against different data sources. I already have NYTimes data source. So, I am looking for another news source ...
11
votes
3answers
4k views

Is a spoken digit dataset available?

Is a labeled dataset of spoken digits, that means of people saying "zero", "one", "two", three", "four", "five", "six", "seven", "eight", or "nine" available? I would also be interested in such a ...
5
votes
1answer
1k views

Longman English Dictionary Database

Is Longman English Dictionary as database available for public or should I digitalise it? I plan to create an app based on Longman English Dictionary, but I need the database somehow. Any idea how?
3
votes
1answer
1k views

Where can I find open etymology databases?

Are there any open and directly accessible, freely available, English etymology dictionary databases like the one used by online etymology dictionary? It could be accessible either directly via ...
9
votes
1answer
756 views

Phones dataset for speech recognition (not telephone number)

In phonetics a phone is a unit of speech sound. I am looking for a dataset of phone waveforms for speech recognition. There are about 17 vowels and 17 consonants for English phones. I tried looking ...
5
votes
1answer
813 views

Where to download the Forvo database?

Forvo.com is a website where native speakers submit pronunciation of words. The audio data is under the BY-NC-SA license. I can't find any obvious place to download the whole data. There is an API, ...
10
votes
3answers
4k views

Dataset of sentences translated into many languages

I'm looking for a dataset of human translated sentences. The ideal dataset would look like this: 1, en, The weather is nice today. 1, de, Das Wetter ist heute schön. 1, es, El clima es agradable hoy....
4
votes
1answer
36 views

Which licence and format should I use for the Norwegian language “data” I am creating?

While learning a foreign language, I am creating my own list of verbs (with their variants when conjugated), list of idioms, nouns (with plural form), etc. As an example, let's focus on the verbs: I ...
15
votes
6answers
4k views

Is there a free list of English word phonetics?

The list actually doesn't have to be free! I just need something that I can buy once or whatever. And if possible, but not needed, parts of speech. Before you say anything, I have googled for this ...
9
votes
6answers
4k views

Dataset of languages and where they are spoken (sub-national)

I am looking for a dataset describing (nearly) all languages, and where they are spoken. Bonus if the areas are linked to Geonames or another mainstream geographical dataset. Polygons are also OK. ...
1
vote
1answer
81 views

List of vocabulary (kanjis) for the JLPT1 exam

I am preparing for the Japanese Language Proficiency Test N1. I am looking for a list of vocabulary (kanjis) that I should probably study for this test. I know there is no official list, but is ...
12
votes
1answer
7k views

Large list of quotes

I am looking for a large database / list of quotes I can use. Preferably famous quotes / quotes by famous people. I have tried looking for a brainyquote API or something, but couldn't find one. If ...
7
votes
2answers
330 views

French equivalent of the brown corpus

The brown corpus is a collection of text where each element is already gramatically tagged. It contains about one million words and is often used to train statistical PoS taggers. For the french ...
4
votes
2answers
523 views

Dataset for Named Entity Recognition on Informal Text

I'm currently searching for labeled datasets to use to train a model to extract named entities from informal text (think something similar to tweets). Because capitalization and grammar are often ...
11
votes
7answers
2k views

List of English words

I'm writing a program that requires a list of English words, such as that used by a spell checker. I have been unable to find any such word lists whatsoever, even after trying 2 search engines.
5
votes
2answers
1k views

Word list for Common European Framework of Reference for Languages (CEFR)

The Common European Framework of Reference for Languages: Learning, Teaching, Assessment, abbreviated as CEFR, is a guideline used to describe achievements of learners of foreign languages across ...
0
votes
1answer
94 views

Database of Japanese words difficulty

I have a huge database of Japanese words and want to split them into flashcard decks of different difficulty levels. (actually for a GSoC project) For this, I need to know the "difficulty" of each ...
11
votes
1answer
3k views

Database of English words difficulty

Data request I am looking for a database that describes the "difficulty" of each English word. Difficulty of a word is not easy to define, but it is probably a mix of: Level of study needed to ...
6
votes
1answer
256 views

Language Similarity Heuristic

I'm looking for a matrix of language pairs that shows some measure of their similarity. My main issue is that I don't know of heuristics that work well across different writing styles. Wikipedia's ...
5
votes
2answers
297 views

Do you know a technique for text mining judicial decisions?

I'm trying to find a method to mine sepecific informations in judicial decisions (judge's name, court, area of law and neutral citation number[canadian way to cite decisions uniformly across the ...
13
votes
1answer
2k views

Speech audio databases with phonemes labelled

Can anyone point me to opensource databases of speech audio with phonemes labelled, .wav or the like + label files, e.g. # time, phoneme 0.31200 ao 0.41200 th 0.54200 er ... I'd prefer English, ...
9
votes
1answer
629 views

Text message corpus for American english

I am doing a research project to identify some patterns in how people interact with each other using text messages. The scope of the project is confined to American users. Can someone suggest a good ...
21
votes
2answers
238 views

Data for word hierarchies

I'm looking for a dataset which will give me is-a and has-a type relations between senses (and give senses from words). WordNet seems to be the most popular model, but I'm trying to evaluate other ...
6
votes
2answers
2k views

Frequency Analysis Character Distribution Data

Is there a location I can download character distributions for frequency analysis used in decryption attempt validation? I am specifically interested in ASCII value [32, 126] frequency distribution ...