I know there is the Wikipedia word frequency lists, but you would have to write a parser for each of those, which is a lot of work, to get the data.
I found these two repos:
- IlyaSemenov/wikipedia-word-frequency: generated the freq from the corresponding Wikipedias written in various languages, so it is its own frequency generated from the various wikipedias. There are about 15 languages.
- hermitdave/FrequencyWords: Many more languages, but not every language by far.
Not sure if that second link is the best we have, or if there is anything else out there. What do you recommend as a way of getting the word frequency lists across many many languages. Perhaps we can even get them for some extinct languages like the Akkadian/Sumerian langs, from ORACC, as they have transcribed texts which could be analyzed. But perhaps extinct languages are too much work, as we don't really have too much content written in them to determine frequency.
Perhaps also there are freely available corpora which could be used to compute frequency, do you know of anything like this if necessary, perhaps? Otherwise, ideally someone has already computed the freq lists for as many languages as possible, so ideally we could just link to that. Ideally we could just download freq lists in one swoop, not one-language-at-a-time. But the fallback datasets (corpora, etc.) would also be useful to know.
