Large English text corpus

Question

I am interested in studying a few specific questions on entropy of different properties of English text. For instance, what is the entropy of capital letters in English? This might tell you something about what letters are more likely to start sentences, or be used in abbreviations or proper nouns.

Does anybody know of a good English text corpus that is readily digestible by a computer program (i.e. plain text) and covers as broad a range of "types" of writing as possible? I am on the fence as to whether I want to focus more on modern English writing or attempt to look at English writing over the last couple hundred years as a whole, so either type of dataset would be fine by me.

Thank you. And please let me know if this belongs on another SE.

Answer 1

Project Gutenberg offers 57.000 free books, available in different formats. Among them, utf-8 encoded plain text with minimal formatting.

The NLTK comes with access to a range of corpora. Among them, a selection from Project Gutenberg, and a chat corpus (if you are looking for more colloquial use of English). Beware of the varying licenses that apply.

If Wikipedia turns out to be a good in your estimation, consider using the WikiExtractor, which can turn a Wikipedia dump into plain text files with minimal formatting.

Answer 2

I have posted a collection of several text datasets related to news here. They contain 50+ mb of text each in an easily parsable format. https://www.kaggle.com/therohk/datasets