Questions tagged [language]

For questions related to human language. Use 'programming' instead for questions related to programming languages.

Filter by
Sorted by
Tagged with
2
votes
1answer
13 views

Where do I find a database of each city in the world and the languages spoken there? (Ex. Barcelona - Catalan, Spanish vs. Madrid - Spanish)

I'm looking to set up a dropdown of languages once the app picks up the city a person is in. Where would I find this database or API?
1
vote
1answer
31 views

Known datasets for long document analysis

Are there good baselines datasets or benchmarks for similarities/retrieval of long texts (extremely long documents, books, etc.)? Although significant both academically and applicational, I cannot get ...
1
vote
0answers
15 views

Label language, how to change to local names

I'm viewing Wikidata with Swedish set as my user interface language. This turns all the language names in the list of labels into Swedish, which makes the list hard to follow in my opinion since the ...
0
votes
2answers
45 views

German nouns gender source

I want to create some sort of spell checking program for German language, for which i need a database with all the words and their gender description(only their gender description) to cross reference ...
5
votes
1answer
131 views

Worldwide holidays, and their names in the local dominant language

Might be slightly OT here but I'll give it a try since the topic is probably well known to many here: We're looking for a database or system to get current and upcoming holidays by location, worldwide,...
1
vote
1answer
139 views

Looking for word frequency data for the Google Books ngrams

I have downloaded all the single words from Google Books ngram data (http://storage.googleapis.com/books/ngrams/books/datasetsv2.html) and I could write code to aggregate it all generate word ...
1
vote
0answers
14 views

Peer to peer communication dataset for research

I am looking for a peer-to-peer communication dataset, chat data, emails, phone calls, etc... any kind of conversation text between pairs of people for research. It would be much useful if the genders(...
0
votes
0answers
14 views

Where can I find data about the present and past number of speakers of the languages of the World?

I'm conducting research on language endangerment and I can't find open data on the number of speakers of different languages. I'm aware that there are some databases on the current number of speakers: ...
9
votes
6answers
5k views

Dataset of languages and where they are spoken (sub-national)

I am looking for a dataset describing (nearly) all languages, and where they are spoken. Bonus if the areas are linked to Geonames or another mainstream geographical dataset. Polygons are also OK. ...
0
votes
0answers
11 views

Language demographics of Swiss citizens

I am looking for demographics of languages spoken or not-spoken by Swiss citizens or residents. Although I don't think it's possible to find this exact dataset, this is the core question: how many ...
1
vote
1answer
38 views

Where can I get word lists for Indian languages, specifically Hindi and Tamil?

Is there a way to get word lists for Indian languages (specifically Tamil and Hindi), even for commercial use? It is for a game I am developing
4
votes
3answers
318 views

Is there any picture corpus?

By "picture corpus" I mean the picture collection each illustrating one word in the usually used vocabulary, such as words in Google 10000 English I found on GitHub. For instance: vocabulary ...
9
votes
5answers
6k views

Database world cities with their names in all languages of the world

I'm looking for a database of cities in the world with their names in all languages of the world. Where it is possible to download?
2
votes
1answer
40 views

Is there a Simplified Chinese thesaurus file, like Moby for English?

I'm looking for a manually curated thesaurus/synonyms file for Simplified Chinese. Like the English Moby thesaurus, but in simplified Chinese. Does such a thing exist? I'm aware that WordNet can be ...
2
votes
2answers
45 views

Linguistics. German verbs

One might think that this question should be on the StackExchange German site, but they appear not to like requests for lists and such like. Hence this post ... I'm looking for large electronic list ...
0
votes
1answer
26 views

English to Telugu and Telugu to English dictionary database

Data: Database of Telugu to English and vice versa language dictionary Context: Need to make a Telugu to English and vice versa dictionary app.There should be quite some public domain dictionaries ...
1
vote
1answer
28 views

Dataset of language families, sub-families and their relations other than Wikidata

I am interested in language families such as Indoeuropean, Romance languages, Afroasiatic and so on, not individual families (although if the dataset includes languages it doesn't hurt). Wikidata and ...
1
vote
0answers
13 views

Grammar for mathematics

I was wondering if there exits an grammar defining mathematical language. By that I mean the list of the tokens used and the syntactic rules.
1
vote
0answers
15 views

Library of ancient Linear A and Linear B text images?

Is there an open-data library of text\language images for ancient Linear A and Linear B languages?
1
vote
2answers
68 views

CEFR for English words by level

Are there any open source data for English words by CEFR level? I found https://www.oxfordlearnersdictionaries.com/wordlists/oxford3000-5000 , which categorizes English words based on CEFR level, ...
6
votes
1answer
1k views

Where to download the Forvo database?

Forvo.com is a website where native speakers submit pronunciation of words. The audio data is under the BY-NC-SA license. I can't find any obvious place to download the whole data. There is an API, ...
1
vote
0answers
13 views

Dictionary of European Languages

I'm looking for a database that contains translations of words in different European languages (mainly Romance). The most important languages for my project are in order Latin, English, Spanish and ...
1
vote
0answers
14 views

translation dictionaries for european languages

I'm looking for open translation dictionaries of european languages. Most important would be english-french and english-spanish. This question is a direct analogon to: Translation dictionaries. The ...
2
votes
1answer
75 views

Mailing list datasets

Related to this question: Obtaining personal mail corpus I'm looking for a list of archived public mailing lists from diverse communities and in diverse languages. For example, a list would look ...
2
votes
1answer
65 views

List of languages with active or native speakers

I'm looking for a simple list that has a count of speakers per language. It can be either active speaker or native speakers, the exactness doesn't matter so much to me. Any format, any license. It's ...
12
votes
2answers
2k views

List of abbreviations and acronyms

I am searching an list of abbreviations,acronyms which should be downloadable as a sql table or json or sth. like that so no api cause it might be not fast enough or has a limit like this one: http://...
5
votes
1answer
396 views

Audio data set of (telephone) conversations and their transcriptions

I need records of telephone conversations (or simple conversations between two persons) with their transcripts. I need English and French datasets.
0
votes
0answers
83 views

Datasets for emotion recognition in German dialogues

I am currently working on the topic of emotion recognition in German dialogues (Speech). So far I only managed to find a single dataset related to this topic: Berlin Database of Emotional Speech. Is ...
2
votes
3answers
120 views

Single-word Translations in All Languages

I am looking for a dataset of single-word translations, essentially a translation dictionary for all languages. I don't need all words, but simply a set of very common words: "yes", "no", "true", "...
1
vote
0answers
34 views

Paragraph Similar Pairs Dataset

I have been working on a project where I need to find whether two paragraphs convey the similar meaning. Same as Quora Question Pairs on Kaggle, but instead of questions, paragraphs. But I could not ...
1
vote
0answers
81 views

Is there a large database of unique surnames and their (origin) language?

Let me explain the phrases I use in my request: large: I'm looking for multiple languages (especially european languages if this constrain helps) and at least 2K surnames per language. But the more ...
0
votes
2answers
101 views

Online Dictionary for Scraping

Does anybody know if there is an extensive online dictionary other than Wiktionary that lists many words on a single page, as opposed to having to search for a single word? None of the online ...
1
vote
1answer
741 views

How to check whether a word is abbreviated or not using Python? [closed]

How do I check whether a word is abbreviated or not in a dataframe column using Python? For instance, I need to detect the value "U.S.A." as an abbreviation. Is there any dictionary present for this?
1
vote
0answers
23 views

Dataset for English word trigrams

Does anyone know where to find such a dataset? There is Google's Ngram viewer but it is too big to be useful. I'm looking for something smaller. Perhaps with only the 100k most common English words ...
2
votes
1answer
33 views

Source of multilingual chat/online/contemporary corpus data?

I need to develop autocorrection and prediction for mobile keyboard app that will work for some 8-10 most common Latin languages + Russian (English, German, Spanish, Portugues, French, ...). For start,...
11
votes
2answers
3k views

American English SMS Text Message Corpora

I'm seeking corpora of American English SMS (<=160 characters) text messages. What corpora are available?
1
vote
0answers
21 views

Estonian named entity recognition data

I'm looking for estonian named entity recognition data. Till now I am unable to find one. Although Estonia has 90% of it's Govt services online, I can't find their NER data anywhere. I tried estnltk, ...
1
vote
0answers
35 views

Historical data of languages spoken in country by percent population?

I am looking for a dataset that contains the languages spoken in each country and their respective percentages of the population, going back as far back into the historical record as possible. I ...
7
votes
1answer
134 views

Corpus of human-scored machine translations?

Most of the parallel corpora (Opus, EuroParl, OpenSubtitles) have only human translations. src_txt | human_trans_of_src_txt_into_trg_lng (We assume all the translations are good.) Is there a corpus ...
3
votes
2answers
362 views

Swedish Noun Gender List

I'm looking for a lexical resource on Swedish noun genders. In my case which nouns that have gender neuter or not, that is if the indefinite article is en or ett.
2
votes
1answer
33 views

Looking for gis data on world linguistics by location

Looking for geospatial information on linguistics. Looking for world wide data with national and subnational data. Data should include what regions speak which language(s). Does this data exist?
0
votes
1answer
21 views

Open dictionary for English with International Phonetic Alphabet for every word

I'm seeking a dictionary that contains the IPA transcription for every word. Is such a resource available for English?
1
vote
0answers
132 views

Where is a large list of declarative and imperative sentences

For a project I need a bunch (+500 each) grammatically correct sentences (English) that are declarative (basic or simple), imperative (command), and interrogative (question) (sorted by group or ...
15
votes
6answers
11k views

Speech audio files dataset with language labels

I am working on building a language classifier in speech/audio samples. I have been trying to find a dataset which may have considerable number of speech samples in various languages. The audio files ...
0
votes
0answers
143 views

Recent dump of names from facebook.com/directory?

For years all names of all Facebook users were listed at https://www.facebook.com/directory. Even without the links to the profiles themselves, it was a very useful dataset of diverse names and ...
1
vote
2answers
59 views

Dataset where words are associated to colors

I am looking for a dataset mapping concrete words with common colors. The only one i've found is this, but it has less than 100 entries. EDIT: example: red_words_list = [ "apple", "blood", "rose", "...
1
vote
0answers
21 views

What are methods to create open text data from copyrighted texts?

Language data are either texts or spoken utterances. By default, any text is copyrighted. We have Open Data for relatively recent time periods, where text were written intentionally with an open ...
6
votes
1answer
69 views

Phonemic & Syllabic N-Gram Distributions of the World's Languages

I am interested in doing some typological analyses by measuring the redundancy and IE entropy across languages in the phonemic and syllabic domains. I've searched for quite a bit but could not find ...
1
vote
1answer
90 views

Find geographic data on the languages spoken in Russia

Does anyone have any ideas for where I can get data on the number or percentage of speakers of a given language in the Russian Federation at some geographic level smaller than federal subject? ...
3
votes
3answers
154 views

For natives of each language, most common second language and proportion of people fluent in it

I am trying to prioritize what languages my software should be translated to. Regardless of other factors, I want to first look at how many people are native speakers of each language, example: ...