I am looking for a machine readable data set (e.g., a mere txt file with one word per row) containing the X most common words in English, excluding proper nouns (unlike this list). If possible, ordered by frequency.
It's not entirely open but the best I've found in this vein is www.wordfrequency.info - I am pretty sure that the 5000 word list is free, with fees for larger lists.
You can also checkout this list at http://www.wordgamedictionary.com/word-lists/common-english-words/
If you need more accurate and representative frequency counts of American English check out the Corpus of Contemporary American English which offers two different-length word lists (60k,100k) that include part-of-speech and frequency distribution across spoken and written registers of speech.