Database of English words difficulty

Question

Data request

I am looking for a database that describes the "difficulty" of each English word.
Difficulty of a word is not easy to define, but it is probably a mix of:

• Level of study needed to understand the signified meaning
• Time it takes for a certain person to read the word
• Probability that a certain person will fail to understand, or misunderstand the word
• Rarity of the signifier in literature
• Length of the signifier
• Rare characters in the signifier
• Rare combinations of characters in the signifier

Context

I have a huge database of English words (sample) and want to split them into flashcard decks of different difficulty levels. (actually for a GSoC project)

That's why I need to know the "difficulty" of each English word.

Note: There are some algorithms to calculate complexity (Flesch Kincaid Reading Ease, Flesch Kincaid Grade Level, Gunning Fog Score, Coleman Liau Index, SMOG Index, Automated Reability Index, Dale-Chall Readability Score, Spache Readability Score, etc) but they are very primitive, and while they can be useful to evaluate the complexity of a whole book, they are totally ineffective for a single word.

Region

Any variant of English, or any combination, is OK

License

Must be reusable and embeddable into my server-side non-redistributed software.

Format

Any format is fine. The most obvious would be a CSV with an integer (or decimal) for each word, like this:

WORD; DIFFICULTY
rhinoceros; 3
cees; 11
leptophyllarum; 14

In any case, I would prefer to avoid web-scraping.

Authority

Preferably data from an academic body, but crowdsourced data is acceptable too.

Requirements

• Must contain at least the 3000 most-used words of the English language
• The complexity scale must have at least 3 different levels, for instance Easy, Medium, Difficult. The more levels the better.

Non-answer

The database at http://elexicon.wustl.edu would be a great answer if it were open data. This project measures how long it takes for a human to do a "lexical decision" for a particular word. So far they have measured 1,123,350 reaction times, by 443 humans, with 40,481 words and 40,481 nonwords. It is not yet clear to me what "lexical decision" actually is, but I am pretty sure it correlates or at least is an important factor in complexity. Unfortunately not open:

Unauthorized Use Strictly Prohibited: Word lists generated from this website are available for non-commercial research purposes only and may not be used in the development of speech technology.

Answer 1

The New General Service List (updated just a few weeks ago) is a list of "the most important words for second language learners of English".

It ranks the top three thousand headwords (plus more words within each headword, such as plurals and tenses).

So it won't cover all the words in your list, and it's really indicating "usefullness" rather than difficulty. But I think useful words are more likely to be used, so more familiar, so "easier". It's certainly something you could use as a major factor, along with other factors, to determine difficulty.

The data is in a spreadsheet, and the license is CC-BY.

For (more difficult) words beyond that list, you could use word frequency as a major component of a difficulty rank. One source is subtitles word frequency lists (CC-BY). That lists a few hundred thousand words by how often they appear in English-language TV and film subtitles. Again not the same as difficulty (and it's spoken English, rather than written), but it's an indication of how likely it is that somebody has heard the word before.

Answer 2

You can simply take a big corpus of the language text, process it, and count the frequency of each word. As a rule of thumb, you can consider more frequent words and being easiest and less frequent ones as being difficult.

There are some processed corpora for this purpose especially for English like TypeDM (https://marcobaroni.org/dm/#data)