How are some companies collecting data for AI projects, for instance openAI (chatGPT), Midjourney, or even those text to video models that will be available soon ? What kind of approaches are they using ? And in general how to collect big amounts of data relatively quick ?
1 Answer
Here is a blog post about common approaches and problems (webz.io)
One common "web-scale datasource" is called "Common Crawl", When downloading it, you might trigger rate limit errors with their IP provider, See status page for past incidents.
A cleaner version of the common crawl is the Collossal Clean Crawled Corpus (C4) dataset, 804 GB on Tensorflow Hub.
