Looking for text data including books, GitHub repositories, webpages, chat logs, and medical, physics, math, computer science, and philosophy papers for Language modeling.
Add a comment
|
2 Answers
The Pile
An 800GB Dataset of Diverse Text for Language Modeling
What is the Pile? The Pile is an 825 GiB diverse, open-source language modeling data set that consists of 22 smaller, high-quality datasets combined together.
Why is the Pile a good training set? Recent work has shown that especially for large models, diversity in data sources improves general cross-domain knowledge of the model, as well as downstream generalization capability. In our evaluations, not only do models trained on the Pile show moderate improvements in traditional language modeling benchmarks, but they also show significant improvements on Pile BPB.
Why is the Pile a good benchmark? To score well on Pile BPB (bits per byte), a model must be able to understand many disparate domains including books, GitHub repositories, webpages, chat logs, and medical, physics, math, computer science, and philosophy papers. Pile BPB is a measure of world knowledge and reasoning ability in these domains, making it a robust benchmark of general, cross-domain text modeling ability for large language models.
wget -m -np -c -U "eye02" -w 2 -R "index.html*" "https://the-eye.eu/public/AI/pile/"
- https://commoncrawl.org/. Larger (petabytes) but more difficult to use than The Pile. More on the size: https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlsize
- OpenWebText
- falcon-refinedweb
answered Feb 5, 2022 at 21:10
