1

I know there is the Wikipedia word frequency lists, but you would have to write a parser for each of those, which is a lot of work, to get the data.

I found these two repos:

Not sure if that second link is the best we have, or if there is anything else out there. What do you recommend as a way of getting the word frequency lists across many many languages. Perhaps we can even get them for some extinct languages like the Akkadian/Sumerian langs, from ORACC, as they have transcribed texts which could be analyzed. But perhaps extinct languages are too much work, as we don't really have too much content written in them to determine frequency.

Perhaps also there are freely available corpora which could be used to compute frequency, do you know of anything like this if necessary, perhaps? Otherwise, ideally someone has already computed the freq lists for as many languages as possible, so ideally we could just link to that. Ideally we could just download freq lists in one swoop, not one-language-at-a-time. But the fallback datasets (corpora, etc.) would also be useful to know.

2 Answers 2

0

https://github.com/rspeer/wordfreq is the best Python library I could find for my uses, it supports 44 languages.

0

On the Google Cloud Platform, on Bigquery-Public-Data, there is the google-books-ngrams-2020 dataset

This Google Books Ngram Dataset contains frequencies of any set of search strings using a yearly count of n-grams found in sources printed between 1500 and 2012 in Google's text corpora.

Languages include English, Chinese (simplified), French, German, Hebrew, Italian, Russian, and Spanish.

The dataset is the same as that behind the Ngram Viewer, but for N=1 (single words only): https://books.google.com/ngrams/info

Primary data can be downloaded here: https://storage.googleapis.com/books/ngrams/books/datasetsv2.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.