7 votes
Accepted

How can I reliably host and share large data sets?

If you want quick, dirty, and free, you can always host downloadable datasets at the Internet Archive (archive.org). No pretty interface, but as much storage as you could ever want.
Asparagirl's user avatar
7 votes

How can I reliably host and share large data sets?

I'm sorry to hear you had a frustrating time uploading to datahub.io and that the site wasn't clear enough about the size limit. Another datahub-like platform that we have worked with in the past is ...
Dan Fowler's user avatar
5 votes

How can I reliably host and share large data sets?

The quick, dirty, easy answer: compress the large data files into multiple .7z files, setting the maximum limit to 95mb per compressed file. I do this to get around GitHub's limitations, and want to ...
albert's user avatar
  • 11.9k
3 votes

Big Data and machine learning to Predictive analytics

"The Unreasonable Effectiveness of Data" is an insightful article, written by by some Google researchers in 2010. If you search for articles that have cited this, you'll find plenty of stuff to read....
knb's user avatar
  • 1,227
3 votes

Real-time data APIs

Here you can find tons of APIs: https://www.programmableweb.com/category/all/apis You also have a filter for streaming APIs: https://www.programmableweb.com/category/streaming/apis?category=20253
kikocorreoso's user avatar
2 votes

How can I reliably host and share large data sets?

I was going to add this as a comment on Dan Fowler's answer but I had to create a new account to post official in my role as co-founder and CPO at data.world. I'm glad you found that data.world's ...
Jon Loyens's user avatar
2 votes

Publicly available transaction data

Here are two sales data sets from Kaggle: Rossmann Store Sales Walmart Recruiting — Store Sales Forecasting You can ask on the forums to get an answer for the dateset licenses from the competition ...
nkhuyu's user avatar
  • 121
1 vote

Where to find an actual bigdata with Variety and Volume?

You can find large public datasets in sources including Google Open Data, Kaggle Datasets, Github and Data.gov. If you are specifically looking for large data coverage, the California Open Data Portal-...
People Data Labs's user avatar
1 vote

How can I flter data from a stream of very large text files?

Have you looked at AWS Athena yet? It doesn't require any additional storage (besides the existing S3 bucket) and you can query the data (all of it) whenever you want - for example if your query ...
philarmour's user avatar
1 vote

How can I reliably host and share large data sets?

Have you thought about Google Spreadsheet? You can retrieve the content of any public Google Spreadsheet in your web app using JSON feeds. The sharing permissions of the Google Spreadsheet should be ...
Marian Paździoch's user avatar
1 vote

Publicly available transaction data

For free sales data, check out this database on Quandl: US Census Bureau - Here are the datasets you'll see if you search for "sales" within this database: https://www.quandl.com/data/USCENSUS-U-S-...
user37317's user avatar
  • 1,036
1 vote

Real-time data APIs

Use Unigraph's data streaming API, it will return you thousands of items per second. Here's how to set it up: https://github.com/unigraph/docs/wiki/Streaming For now just run this sample query in ...
marfi's user avatar
  • 583
1 vote

open Big Data to solve cancer epidemiology challenge

If you're interested in brain imaging data, you could check out the Connectome Coordination Facility I downloaded the original HCP 1200 for our institution a couple of years free of charge. That's 7T ...
Matt's user avatar
  • 211
1 vote

What's the difference between Open Data and Big Data

Big Data. I think the point has been missed about "Big Data". It doesn't need to be large in volume. There are historically (from 2001) three tenets of big data high volume (e.g. Facebook, Google ...
Marcus D's user avatar
  • 1,119

Only top scored, non community-wiki answers of a minimum length are eligible