9

Relevant to this answer, I discovered that datahub.io has a 100MB limit on file uploads. I was disappointed to learn this, and it tooks many server errors to realize I should read the FAQ, where I learned there is a limit.

The file upload size limit for datahub.io is set to 100MB. If your files are bigger than that you can host them on a separate storage provider like Dropbox or Amazon S3 and link to them from your datahub.io dataset.

Github has a similar 100MB filesize limit.

I'm not opposed to paying to host large files,

  • but how can I share big datasets that are not personally connected to me, like Dropbox? (S3 isn't really an option to host 1 file)

  • and that offer a one-time fee to host the file "forever".

In this particular case, the zipped CSV is 440MB, so not TB or even GBs of data.


Related, but without the focus on large files:

How do I share Open Data with others on this SE site?

Note: torrent could work, but requires a file that doesn't change and also requires decent interest from the community. For this question I'd like to focus on http/ftp hosts.

  • You might look at modeanalytics.com -- I do this on my own server (SQL access to several databases, see wordpress.barrycarter.info for a list [top menu under "DB"]), and would love to see a great free hosted solution that didn't require using my own space/bandwidth. – Barry Carter May 18 '17 at 21:11
7

I'm sorry to hear you had a frustrating time uploading to datahub.io and that the site wasn't clear enough about the size limit. Another datahub-like platform that we have worked with in the past is data.world. Their size limit is 500MB.

  • 1
    nice, i almost shared this with data.world's team in slack. will do now, maybe they want to respond. – albert May 18 '17 at 12:51
  • 1
    I moved my NEISS datasets there, and will accept this answer in a couple days. One file is 700+MB zipped, and they still allowed the upload. Thanks! – philshem May 18 '17 at 14:22
  • i love seeing things like this. its like we're a team! not at all...but totally! – albert May 18 '17 at 18:53
5

If you want quick, dirty, and free, you can always host downloadable datasets at the Internet Archive (archive.org). No pretty interface, but as much storage as you could ever want.

4

The quick, dirty, easy answer: compress the large data files into multiple .7z files, setting the maximum limit to 95mb per compressed file. I do this to get around GitHub's limitations, and want to say I've done it on datahub.io too, though I can't recall exactly. I have done it on CKAN instances, so I guess that covers it.
Note: GLFS (Git Large File Storage), but its limits are also not clearly displayed; I learned about them after hitting 5gb limit.

Possible Solutions:

Dat Project was made exactly for sharing large sets; not a hosting solution, but does let the data get out of the silos we keep locally. Dat also recently got Knight Foundation funding towards publicly hosting select large datasets. It might be worthwhile to reach out to them and see if they would be interested in hosting it.

Wolfram Alpha has launched a Data Repository that looks incredibly interesting and in-depth; unfortunately it currently (2017-05-19) responds to the submit data tab with a coming soon message. So not much to share, not even sure if its open, but certainly something to watch.

2

I was going to add this as a comment on Dan Fowler's answer but I had to create a new account to post official in my role as co-founder and CPO at data.world. I'm glad you found that data.world's upload to work for you. We do cap datasets at 500mb of structured data that we recognize (JSON, CSV, TSV, TTL, N-Triples, Sqllite, Excel). The reason for that is that data is converted into RDF triples and loaded into a graph DB that you can query (TLDR; upload a CSV, JSON or Excel... query them all with SQL or SPARQL and even do federated queries across datasets). For binary or unstructured files that we don't know how to parse yet, have at it. You can also email our support group and get dataset size limits increased.

1

Have you thought about Google Spreadsheet?

You can retrieve the content of any public Google Spreadsheet in your web app using JSON feeds. The sharing permissions of the Google Spreadsheet should be either “Public” or set to “Anyone with link can view” for the app to fetch cells from the Google Spreadsheet without authentication.

You will also need to publish the sheet (File -> Publish to the web -> Publish) for the data to be accessible from your website or REST powered web app.

The JSON and XML feeds for any Google Spreadsheet is available at:

JSON Format: https://spreadsheets.google.com/feeds/list/SPREADSHEET/od6/public/basic?alt=json

(answer credits: stolen from here)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.