Data request
I am looking for a database that describes the "difficulty" of each English word.
Difficulty of a word is not easy to define, but it is probably a mix of:
- Level of study needed to understand the signified meaning
- Time it takes for a certain person to read the word
- Probability that a certain person will fail to understand, or misunderstand the word
- Rarity of the signifier in literature
- Length of the signifier
- Rare characters in the signifier
- Rare combinations of characters in the signifier
Context
I have a huge database of English words (sample) and want to split them into flashcard decks of different difficulty levels. (actually for a GSoC project)
That's why I need to know the "difficulty" of each English word.
Note: There are some algorithms to calculate complexity (Flesch Kincaid Reading Ease, Flesch Kincaid Grade Level, Gunning Fog Score, Coleman Liau Index, SMOG Index, Automated Reability Index, Dale-Chall Readability Score, Spache Readability Score, etc) but they are very primitive, and while they can be useful to evaluate the complexity of a whole book, they are totally ineffective for a single word.
Region
Any variant of English, or any combination, is OK
License
Must be reusable and embeddable into my server-side non-redistributed software.
Format
Any format is fine. The most obvious would be a CSV with an integer (or decimal) for each word, like this:
WORD; DIFFICULTY
rhinoceros; 3
cees; 11
leptophyllarum; 14
In any case, I would prefer to avoid web-scraping.
Authority
Preferably data from an academic body, but crowdsourced data is acceptable too.
Requirements
- Must contain at least the 3000 most-used words of the English language
- The complexity scale must have at least 3 different levels, for instance Easy, Medium, Difficult. The more levels the better.
Non-answer
The database at http://elexicon.wustl.edu would be a great answer if it were open data. This project measures how long it takes for a human to do a "lexical decision" for a particular word. So far they have measured 1,123,350 reaction times, by 443 humans, with 40,481 words and 40,481 nonwords. It is not yet clear to me what "lexical decision" actually is, but I am pretty sure it correlates or at least is an important factor in complexity. Unfortunately not open:
Unauthorized Use Strictly Prohibited: Word lists generated from this website are available for non-commercial research purposes only and may not be used in the development of speech technology.