2

I am interested in studying a few specific questions on entropy of different properties of English text. For instance, what is the entropy of capital letters in English? This might tell you something about what letters are more likely to start sentences, or be used in abbreviations or proper nouns.

Does anybody know of a good English text corpus that is readily digestible by a computer program (i.e. plain text) and covers as broad a range of "types" of writing as possible? I am on the fence as to whether I want to focus more on modern English writing or attempt to look at English writing over the last couple hundred years as a whole, so either type of dataset would be fine by me.

Thank you. And please let me know if this belongs on another SE.

2 Answers 2

3

Project Gutenberg offers 57.000 free books, available in different formats. Among them, utf-8 encoded plain text with minimal formatting.

The NLTK comes with access to a range of corpora. Among them, a selection from Project Gutenberg, and a chat corpus (if you are looking for more colloquial use of English). Beware of the varying licenses that apply.

If Wikipedia turns out to be a good in your estimation, consider using the WikiExtractor, which can turn a Wikipedia dump into plain text files with minimal formatting.

1

I have posted a collection of several text datasets related to news here. They contain 50+ mb of text each in an easily parsable format. https://www.kaggle.com/therohk/datasets

2
  • Did you post material with copyright in there?
    – user4293
    Aug 22, 2019 at 7:55
  • the files contain publicly available information only and available under cc0 lisence. no strings attached.
    – Rohit
    Mar 9, 2020 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.