1
The Pile
An 800GB Dataset of Diverse Text for Language Modeling
What is the Pile?
The Pile is an 825 GiB diverse, open-source language modeling data set that consists of 22 smaller, high-quality datasets combined together.
Why is the Pile a good training set?
Recent work has shown that especially for large models, diversity in data sources improves general ...
Only top voted, non community-wiki answers of a minimum length are eligible
