Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [language]

For questions related to human language. Use 'programming' instead for questions related to programming languages.

Filter by
Sorted by
Tagged with
1 vote
2 answers
75 views

Word frequency dataset across many languages?

I know there is the Wikipedia word frequency lists, but you would have to write a parser for each of those, which is a lot of work, to get the data. I found these two repos: IlyaSemenov/wikipedia-...
Lance's user avatar
  • 387
0 votes
0 answers
51 views

Dataset of geographic distribution of languages and/or language families across the globe?

Where can one find a free or open dataset listing the geographic boundaries of languages and/or language families, something like this (but the raw JSON/XML/CSV data behind it): Would like to show an ...
Lance's user avatar
  • 387
2 votes
1 answer
35 views

List of alphabet and other character data per language as structured data?

I know the unicode dataset has information about "this character has numerical value X", and such, and it also has sometimes a sort order for the characters. But I don't think it has ...
Lance's user avatar
  • 387
1 vote
1 answer
118 views

A file of all Korean language words?

Is there an open data set file (not an open API) containing all (or almost all) Korean language words? There are some famous Korean dictionaries such as the Standard Korean Dictionary published by the ...
user67275's user avatar
  • 121
2 votes
1 answer
35 views

Speech intelligiblity tests such as rhyme tests

Can anyoe point me to .wav files or the like for speech intelligiblity tests, e.g. Rhyme tests "not, tot, got, pot, hot, lot" ? What I want are reproducible tests for various hearing aids ...
denis's user avatar
  • 468
1 vote
0 answers
63 views

Words list by category / topic

For my side project, I need to get a categorization of as many English words as possible. With "categorization of words" I mean the assignment of categories / topics to each word, as it is ...
Bass's user avatar
  • 111
1 vote
0 answers
26 views

Data for multi-modal semantic search

Context: I am interested in the potential to use vector indexing and semantic search for multimodal data. For instance, a retail website may have unstructured text data from the advertising copy on ...
greggs's user avatar
  • 111
1 vote
1 answer
388 views

Looking for a free dataset of book titles classified by some well-established and unified classification system

I am looking for a free dataset of book titles: Classified by Dewey Decimal Classification (DDC) or by Library of Congress Classification (LCC). Wide coverage. I've seen Researcher Format (CSV) ...
dzieciou's user avatar
  • 233
0 votes
2 answers
56 views

French list of word where I can use filters like nouns, singular form, verbs…

I often need to find a list of french words (sometimes english as well, but that's certainly for another question) where I can filter words depending on multiple criterias. For instance, I may like to ...
tobiasBora's user avatar
1 vote
1 answer
87 views

Dataset for human-level artificial intelligence

I think a human-centric multi-modal dataset (i.e.: head mounted binocular cameras, steroes mics, text transciption, skeletal pose, skin area contact) would be a great starting point for building human-...
Jacob Valdez's user avatar
0 votes
1 answer
37 views

Extemporaneous speech, as text, tagged by speaker?

tl;dr; I'm looking for a corpus (or multiple of them) of un-scripted English speech, transcribed to text and tagged by who said what. Ideally, I'd like hundreds or even thousands of distinct speakers. ...
BCS's user avatar
  • 101
1 vote
2 answers
309 views

Russian Dictionary that contains stress marks

I am looking to automate the process of setting the stress of Russian words. There are already tools that do this (RussianGram or https://morpher.ru/accentizer/), but nothing open source which can be ...
Pux's user avatar
  • 171
1 vote
0 answers
96 views

Dataset with French words by CEFR level

As a German speaker learning French, I really enjoyed the "Fremdsprachentexte Reclam" books: they are French literature books for German learners, each book has a CEFR level (e.g. B2, C1...),...
johnnewman's user avatar
1 vote
0 answers
41 views

How many *Non-Native* speakers of Egyptian Arabic are there world wide?

Every time I try to look this up, it always gives me the number for native speakers. I'm not looking for that. I specifically want to know how prevalent Egyptian Arabic is as a second language around ...
RexxiA's user avatar
  • 11
3 votes
1 answer
120 views

Where do I find a database of each city in the world and the languages spoken there? (Ex. Barcelona - Catalan, Spanish vs. Madrid - Spanish)

I'm looking to set up a dropdown of languages once the app picks up the city a person is in. Where would I find this database or API?
Jeyashree Haridoss's user avatar
1 vote
0 answers
27 views

Label language, how to change to local names

I'm viewing Wikidata with Swedish set as my user interface language. This turns all the language names in the list of labels into Swedish, which makes the list hard to follow in my opinion since the ...
Sabelöga's user avatar
0 votes
1 answer
27 views

Language demographics of Swiss citizens

I am looking for demographics of languages spoken or not-spoken by Swiss citizens or residents. Although I don't think it's possible to find this exact dataset, this is the core question: how many ...
philshem's user avatar
  • 17.6k
2 votes
1 answer
61 views

Is there a Simplified Chinese thesaurus file, like Moby for English?

I'm looking for a manually curated thesaurus/synonyms file for Simplified Chinese. Like the English Moby thesaurus, but in simplified Chinese. Does such a thing exist? I'm aware that WordNet can be ...
Simon Hardman's user avatar
2 votes
2 answers
72 views

Linguistics. German verbs

One might think that this question should be on the StackExchange German site, but they appear not to like requests for lists and such like. Hence this post ... I'm looking for large electronic list ...
CrimsonDark's user avatar
1 vote
0 answers
27 views

Peer to peer communication dataset for research

I am looking for a peer-to-peer communication dataset, chat data, emails, phone calls, etc... any kind of conversation text between pairs of people for research. It would be much useful if the genders(...
Jean's user avatar
  • 41
0 votes
2 answers
262 views

German nouns gender source

I want to create some sort of spell checking program for German language, for which i need a database with all the words and their gender description(only their gender description) to cross reference ...
E.K's user avatar
  • 9
1 vote
1 answer
82 views

Dataset of language families, sub-families and their relations other than Wikidata

I am interested in language families such as Indoeuropean, Romance languages, Afroasiatic and so on, not individual families (although if the dataset includes languages it doesn't hurt). Wikidata and ...
alotropico's user avatar
1 vote
0 answers
14 views

Grammar for mathematics

I was wondering if there exits an grammar defining mathematical language. By that I mean the list of the tokens used and the syntactic rules.
Sevla's user avatar
  • 11
1 vote
0 answers
16 views

Library of ancient Linear A and Linear B text images?

Is there an open-data library of text\language images for ancient Linear A and Linear B languages?
DuckQueen's user avatar
  • 357
2 votes
2 answers
2k views

CEFR for English words by level

Are there any open source data for English words by CEFR level? I found https://www.oxfordlearnersdictionaries.com/wordlists/oxford3000-5000 , which categorizes English words based on CEFR level, ...
Saber's user avatar
  • 71
3 votes
1 answer
245 views

English to Telugu and Telugu to English dictionary database

Data: Database of Telugu to English and vice versa language dictionary Context: Need to make a Telugu to English and vice versa dictionary app.There should be quite some public domain dictionaries ...
Flutty Dev's user avatar
1 vote
0 answers
23 views

Dictionary of European Languages

I'm looking for a database that contains translations of words in different European languages (mainly Romance). The most important languages for my project are in order Latin, English, Spanish and ...
user avatar
1 vote
0 answers
30 views

translation dictionaries for european languages

I'm looking for open translation dictionaries of european languages. Most important would be english-french and english-spanish. This question is a direct analogon to: Translation dictionaries. The ...
lhk's user avatar
  • 133
1 vote
2 answers
61 views

Where can I get word lists for Indian languages, specifically Hindi and Tamil?

Is there a way to get word lists for Indian languages (specifically Tamil and Hindi), even for commercial use? It is for a game I am developing
user21861's user avatar
0 votes
0 answers
147 views

Datasets for emotion recognition in German dialogues

I am currently working on the topic of emotion recognition in German dialogues (Speech). So far I only managed to find a single dataset related to this topic: Berlin Database of Emotional Speech. Is ...
SuperKogito's user avatar
1 vote
0 answers
46 views

Paragraph Similar Pairs Dataset

I have been working on a project where I need to find whether two paragraphs convey the similar meaning. Same as Quora Question Pairs on Kaggle, but instead of questions, paragraphs. But I could not ...
AJAY HAYAGREEVE's user avatar
1 vote
0 answers
204 views

Is there a large database of unique surnames and their (origin) language?

Let me explain the phrases I use in my request: large: I'm looking for multiple languages (especially european languages if this constrain helps) and at least 2K surnames per language. But the more ...
Marv's user avatar
  • 111
0 votes
3 answers
409 views

Online Dictionary for Scraping

Does anybody know if there is an extensive online dictionary other than Wiktionary that lists many words on a single page, as opposed to having to search for a single word? None of the online ...
oldboy's user avatar
  • 121
1 vote
0 answers
45 views

Dataset for English word trigrams

Does anyone know where to find such a dataset? There is Google's Ngram viewer but it is too big to be useful. I'm looking for something smaller. Perhaps with only the 100k most common English words ...
Gaslight Deceive Subvert's user avatar
2 votes
1 answer
52 views

Source of multilingual chat/online/contemporary corpus data?

I need to develop autocorrection and prediction for mobile keyboard app that will work for some 8-10 most common Latin languages + Russian (English, German, Spanish, Portugues, French, ...). For start,...
Rasto's user avatar
  • 121
1 vote
0 answers
26 views

Estonian named entity recognition data

I'm looking for estonian named entity recognition data. Till now I am unable to find one. Although Estonia has 90% of it's Govt services online, I can't find their NER data anywhere. I tried estnltk, ...
Hasan Shaukat's user avatar
2 votes
0 answers
65 views

Historical data of languages spoken in country by percent population?

I am looking for a dataset that contains the languages spoken in each country and their respective percentages of the population, going back as far back into the historical record as possible. I ...
Jeremy Dohmann's user avatar
1 vote
1 answer
2k views

How to check whether a word is abbreviated or not using Python? [closed]

How do I check whether a word is abbreviated or not in a dataframe column using Python? For instance, I need to detect the value "U.S.A." as an abbreviation. Is there any dictionary present for this?
user avatar
2 votes
1 answer
46 views

Looking for gis data on world linguistics by location

Looking for geospatial information on linguistics. Looking for world wide data with national and subnational data. Data should include what regions speak which language(s). Does this data exist?
A. S.'s user avatar
  • 51
1 vote
1 answer
512 views

Looking for word frequency data for the Google Books ngrams

I have downloaded all the single words from Google Books ngram data (http://storage.googleapis.com/books/ngrams/books/datasetsv2.html) and I could write code to aggregate it all generate word ...
Zaradaqaw's user avatar
1 vote
0 answers
235 views

Where is a large list of declarative and imperative sentences

For a project I need a bunch (+500 each) grammatically correct sentences (English) that are declarative (basic or simple), imperative (command), and interrogative (question) (sorted by group or ...
Jack's user avatar
  • 111
0 votes
0 answers
208 views

Recent dump of names from facebook.com/directory?

For years all names of all Facebook users were listed at https://www.facebook.com/directory. Even without the links to the profiles themselves, it was a very useful dataset of diverse names and ...
Adam Bittlingmayer's user avatar
1 vote
0 answers
27 views

What are methods to create open text data from copyrighted texts?

Language data are either texts or spoken utterances. By default, any text is copyrighted. We have Open Data for relatively recent time periods, where text were written intentionally with an open ...
user avatar
1 vote
2 answers
95 views

Open dictionary for English with International Phonetic Alphabet for every word

I'm seeking an openly-licensed dictionary that contains for every word the IPA transcription. Is such a resource available for English?
Christian's user avatar
  • 661
2 votes
1 answer
103 views

List of languages with active or native speakers

I'm looking for a simple list that has a count of speakers per language. It can be either active speaker or native speakers, the exactness doesn't matter so much to me. Any format, any license. It's ...
philshem's user avatar
  • 17.6k
5 votes
1 answer
2k views

Audio data set of (telephone) conversations and their transcriptions

I need records of telephone conversations (or simple conversations between two persons) with their transcripts. I need English and French datasets.
K.cyrine's user avatar
1 vote
1 answer
100 views

Find geographic data on the languages spoken in Russia

Does anyone have any ideas for where I can get data on the number or percentage of speakers of a given language in the Russian Federation at some geographic level smaller than federal subject? ...
Andi Qu's user avatar
  • 111
2 votes
2 answers
160 views

Dataset where words are associated to colors

I am looking for a dataset mapping concrete words with common colors. The only one i've found is this, but it has less than 100 entries. EDIT: example: red_words_list = [ "apple", "blood", "rose", "...
eadmaster's user avatar
  • 121
3 votes
2 answers
586 views

Swedish Noun Gender List

I'm looking for a lexical resource on Swedish noun genders. In my case which nouns that have gender neuter or not, that is if the indefinite article is en or ett.
Nordlöw's user avatar
  • 261
1 vote
0 answers
32 views

Hierarchical semantic distances dataset

I need data (preferably human judgment) for the structural/hierarchical semantic distance between many couples (at least hundreds) of words. For example, d(computer, television) < d(radio, ...
Cranjis's user avatar
  • 111