An 800GB Dataset of Diverse Text for Language Modeling
What is the Pile?
The Pile is an 825 GiB diverse, open-source language modeling data set that consists of 22 smaller, high-quality datasets combined together.
Why is the Pile a good training set?
Recent work has shown that especially for large models, diversity in data sources improves general ...