This question already has an answer here:
I have a project. It's basically doing 2 things:
- collect data
- do something with the data
The codeI use is open source and available on GitHub.
The crawler that collects the dataset works slow (400 pages per minute), and the resulting dataset will big: 40 million records, more than 2000 MB.
I assume that the dataset could be useful as a stand-alone object. May be somebody will do something else with the data and the person will not have wait 100 days to collect it again, if I share it.
Where can I store this data, to make it available for any person and to keep it free of charge without abusing GitHub?
History of the dataset changing would be a nice feature...