3

I have a project. It's basically doing 2 things:
- collect data
- do something with the data

The codeI use is open source and available on GitHub.

The crawler that collects the dataset works slow (400 pages per minute), and the resulting dataset will big: 40 million records, more than 2000 MB.

I assume that the dataset could be useful as a stand-alone object. May be somebody will do something else with the data and the person will not have wait 100 days to collect it again, if I share it.

Where can I store this data, to make it available for any person and to keep it free of charge without abusing GitHub?

History of the dataset changing would be a nice feature...

|improve this question|||||
3

What makes you think storing data on GitHub is an abuse of GitHub?
It most certainly is not. GitHub showcases Open Data repositories as one of its features.
So use GitHub, you already are anyways.
If GitHub simply will not do, check out datahub.io, data.world, or data portals that are specific to the type(s) of data you have been collecting.

If size is the issue and you aren't comfortable with breaking the data up into multiple zips to get around GitHub's sizing issues, then Dat data and Beaker Browser seem like another possible solution for you.

|improve this answer|||||
  • help.github.com/articles/working-with-large-files "We suggest removing the following types of files: Database dumps" and "Conditions for large files: GitHub will warn you when pushing files larger than 50 MB. You will not be allowed to push files larger than 100 MB". – Areso Jun 10 '18 at 15:53
  • And zipping it into 20 pieces it's act of abuse as for me. May be I should try Github LFS, though IDK whether it works with non-paid accounts. – Areso Jun 10 '18 at 16:00
  • data.world is no go with their prices and free options covered with GitHub. Datahub, though, looks more interesting in terms of free options. – Areso Jun 10 '18 at 16:10
  • zipping into 20 pieces is act of abuse for you...that is what i do. its not against the rules. i think glfs has a free tier for a little bit of data, but then kicks in pricing. – albert♦ Jun 10 '18 at 16:44
1

On other site I get a handful of responses.
I pick one of them:
use public clouds, such as skydrive, google drive, yandex disk, mail.ru cloud, mega or anything else.
From all of them I picked up Yandex Disk, because they provide webdav option (REST API also available) to get it integrated into any project just with
curl -T ./local_file.zip --user "{username}:{password}" https://webdav.yandex.ru/backup.zip
(this solution provided by Anatoliy Skornyakov)
So, I think it's great idea to use a public cloud with a great amount of space provided for free of charge.

|improve this answer|||||

Not the answer you're looking for? Browse other questions tagged or ask your own question.