2

I am looking for a dataset of single-word translations, essentially a translation dictionary for all languages. I don't need all words, but simply a set of very common words: "yes", "no", "true", "false", "zero", "one", etc. It also doesn't have to be every language, though more languages the better. If the dataset comes with ISO 639 codes that would be even better.

This question is related, but it is asking for whole sentence translations, whereas I'm simply looking for individual word translations. The accepted answer there points to tatoeba.org, which might work for my needs, but I'm hoping to find other options to consider.

Improve this question

3 Answers 3

Reset to default
5

I'm not aware of an available dataset in the requested form, but it should be possible to obtain something similar by parsing data from wiktionary. The latest dump can be obtained at https://dumps.wikimedia.org/enwiktionary/latest/

Also, some interesting links can be found by googling "wiktionary rdf"

Improve this answer
2

If you want to find which words are most common you can scan 1 billion web pages from 2009 in 10 languages. It's 5TB compressed. They have subsets of data too.

Common Crawl web pages are from Oct 2015 and previous years.

More web page collections here at Rockstar datasets on Github.

Improve this answer
0

If a Swadesh list is good enough for common words, then you probably want to use Panlex.

https://panlex.org/snapshot/

Improve this answer
2
  • what is "Swadesh"?
    – albert
    Aug 12, 2019 at 22:42
  • 1
    Swadesh list is a list of words that "tend not to change" over time for languages. As a result they also have a high word frequencies
    – hussaibi
    Aug 13, 2019 at 3:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.