How can I reliably host and share large data sets?

Question

Relevant to this answer, I discovered that datahub.io has a 100MB limit on file uploads. I was disappointed to learn this, and it tooks many server errors to realize I should read the FAQ, where I learned there is a limit.

The file upload size limit for datahub.io is set to 100MB. If your files are bigger than that you can host them on a separate storage provider like Dropbox or Amazon S3 and link to them from your datahub.io dataset.

Github has a similar 100MB filesize limit.

I'm not opposed to paying to host large files,

• but how can I share big datasets that are not personally connected to me, like Dropbox? (S3 isn't really an option to host 1 file)

• and that offer a one-time fee to host the file "forever".

In this particular case, the zipped CSV is 440MB, so not TB or even GBs of data.

---

Related, but without the focus on large files:

How do I share Open Data with others on this SE site?

Note: torrent could work, but requires a file that doesn't change and also requires decent interest from the community. For this question I'd like to focus on http/ftp hosts.

Answer 1

If you want quick, dirty, and free, you can always host downloadable datasets at the Internet Archive (archive.org). No pretty interface, but as much storage as you could ever want.

Answer 2

I'm sorry to hear you had a frustrating time uploading to datahub.io and that the site wasn't clear enough about the size limit. Another datahub-like platform that we have worked with in the past is data.world. Their size limit is 500MB.

Answer 3

The quick, dirty, easy answer: compress the large data files into multiple .7z files, setting the maximum limit to 95mb per compressed file. I do this to get around GitHub's limitations, and want to say I've done it on datahub.io too, though I can't recall exactly. I have done it on CKAN instances, so I guess that covers it.
Note: GLFS (Git Large File Storage), but its limits are also not clearly displayed; I learned about them after hitting 5gb limit.

Possible Solutions:

Dat Project was made exactly for sharing large sets; not a hosting solution, but does let the data get out of the silos we keep locally. Dat also recently got Knight Foundation funding towards publicly hosting select large datasets. It might be worthwhile to reach out to them and see if they would be interested in hosting it.

Wolfram Alpha has launched a Data Repository that looks incredibly interesting and in-depth; unfortunately it currently (2017-05-19) responds to the submit data tab with a coming soon message. So not much to share, not even sure if its open, but certainly something to watch.

Answer 4

I was going to add this as a comment on Dan Fowler's answer but I had to create a new account to post official in my role as co-founder and CPO at data.world. I'm glad you found that data.world's upload to work for you. We do cap datasets at 500mb of structured data that we recognize (JSON, CSV, TSV, TTL, N-Triples, Sqllite, Excel). The reason for that is that data is converted into RDF triples and loaded into a graph DB that you can query (TLDR; upload a CSV, JSON or Excel... query them all with SQL or SPARQL and even do federated queries across datasets). For binary or unstructured files that we don't know how to parse yet, have at it. You can also email our support group and get dataset size limits increased.

Answer 5

Have you thought about Google Spreadsheet?

You can retrieve the content of any public Google Spreadsheet in your web app using JSON feeds. The sharing permissions of the Google Spreadsheet should be either “Public” or set to “Anyone with link can view” for the app to fetch cells from the Google Spreadsheet without authentication.

You will also need to publish the sheet (File -> Publish to the web -> Publish) for the data to be accessible from your website or REST powered web app.

The JSON and XML feeds for any Google Spreadsheet is available at:

JSON Format: https://spreadsheets.google.com/feeds/list/SPREADSHEET/od6/public/basic?alt=json

(answer credits: stolen from here)

Answer 6

A few months ago I stumbled upon this question and realised that no solution existed.

I have since built a platform to solve this problem for myself, and figured it could benefit others: Joule (discontinued as of November 2022)

I tried to address the following problems:

• no file size limit
• data compression
• easy integration into terminal-workflows

Answer 7

Github releases have a maximum single file size of 2 GB (and have no size limits for the entire release, so you can upload many files). The result is also directly linkable so that you can automatically download it from a script.