I am looking for a machine readable data set (e.g., a mere txt file with one word per row) containing the X most common words in English, excluding proper nouns (unlike this list). If possible, ordered by frequency.
asked Mar 11, 2016 at 6:19
-
1You can get the top ten hundred words from xkcd.com/simplewriter/words.js ...I thought it had duplicates (I'm, I'd, I'll), but it looks like he has all contractions w/ two different types of apostrophes.– JoeCommented Mar 11, 2016 at 14:54
-
1Because English capitalizes proper nouns and not all nouns, like German, it's fairly easy to do this with a few lines of code. This example does it for characters, but the switch to "words that don't start with a capital letter, except I" is easy enough opendata.stackexchange.com/a/7043/1511– philshemCommented Mar 12, 2016 at 7:24
Add a comment
|
5 Answers
It's not entirely open but the best I've found in this vein is www.wordfrequency.info - I am pretty sure that the 5000 word list is free, with fees for larger lists.
There are word frequency lists for a number of languages, including English, that were created by Hermit Dave from the data at opensubtitles.org.
You can download them for free on his blog. The licence is: Creative Commons – Attribution / ShareAlike 3.0 license.
You can also checkout this list at http://www.wordgamedictionary.com/word-lists/common-english-words/
If you need more accurate and representative frequency counts of American English check out the Corpus of Contemporary American English which offers two different-length word lists (60k,100k) that include part-of-speech and frequency distribution across spoken and written registers of speech.
answered Apr 15, 2016 at 22:59
See resources at Wiktionary:Frequency Lists
In particular I have used Leipzig Corpora Collection. Use cut -f2 if you just want the words and not the rank or frequency.
