I am looking for a machine readable data set (e.g., a mere txt file with one word per row) containing the X most common words in English, excluding proper nouns (unlike this list). If possible, ordered by frequency.

  • 1
    You can get the top ten hundred words from xkcd.com/simplewriter/words.js ...I thought it had duplicates (I'm, I'd, I'll), but it looks like he has all contractions w/ two different types of apostrophes.
    – Joe
    Commented Mar 11, 2016 at 14:54
  • 1
    Because English capitalizes proper nouns and not all nouns, like German, it's fairly easy to do this with a few lines of code. This example does it for characters, but the switch to "words that don't start with a capital letter, except I" is easy enough opendata.stackexchange.com/a/7043/1511
    – philshem
    Commented Mar 12, 2016 at 7:24

5 Answers 5


It's not entirely open but the best I've found in this vein is www.wordfrequency.info - I am pretty sure that the 5000 word list is free, with fees for larger lists.


There are word frequency lists for a number of languages, including English, that were created by Hermit Dave from the data at opensubtitles.org.

You can download them for free on his blog. The licence is: Creative Commons – Attribution / ShareAlike 3.0 license.


You can also checkout this list at http://www.wordgamedictionary.com/word-lists/common-english-words/


If you need more accurate and representative frequency counts of American English check out the Corpus of Contemporary American English which offers two different-length word lists (60k,100k) that include part-of-speech and frequency distribution across spoken and written registers of speech.
enter image description here


See resources at Wiktionary:Frequency Lists

In particular I have used Leipzig Corpora Collection. Use cut -f2 if you just want the words and not the rank or frequency.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.