8

Here you are: A MOOC from coursera, but you have to wait until they will run it again or You can start from dozens of courses here in big data university.


7

I'm sorry to hear you had a frustrating time uploading to datahub.io and that the site wasn't clear enough about the size limit. Another datahub-like platform that we have worked with in the past is data.world. Their size limit is 500MB.


6

If you want quick, dirty, and free, you can always host downloadable datasets at the Internet Archive (archive.org). No pretty interface, but as much storage as you could ever want.


5

The quick, dirty, easy answer: compress the large data files into multiple .7z files, setting the maximum limit to 95mb per compressed file. I do this to get around GitHub's limitations, and want to say I've done it on datahub.io too, though I can't recall exactly. I have done it on CKAN instances, so I guess that covers it. Note: GLFS (Git Large File ...


5

I also think that Twitter data is the best way to get >100Gb french dataset. OpenStreetMaps gives really great open data but you can't reach 100Gb easily : the Integral Metropolitan France dataset is 2.6 GB. There is no open weather data by MeteoFrance (you have to pay the licence) but there is an Open Meteo Forecast that you can try. I never had the ...


4

You're probably going to need to get information from many jurisdictions - at least in the US, there is not a good/granular sources for property tax, crime, and other information for all states in one dataset. For crime data, check out FBI UCR reports which are usually not granular. Each city may (hopefully) report their own data. Try google and/or http://...


4

In addition to educational materials, Cloudera offers a free Hadoop cluster that is ready to go. After installing VirtualBox, KVM or VMWare, you can download and run the virtual machine. link blog post youtube tutorial Hortonworks also offers a complete VM: "sandbox. This comes with Hive, Pig, and other tools that can be managed by the browser frontend. ...


3

"The Unreasonable Effectiveness of Data" is an insightful article, written by by some Google researchers in 2010. If you search for articles that have cited this, you'll find plenty of stuff to read.


3

Here you can find tons of APIs: https://www.programmableweb.com/category/all/apis You also have a filter for streaming APIs: https://www.programmableweb.com/category/streaming/apis?category=20253


3

At least a million records you say? I know of a data set that could fit the bill. The 2013 ACS 5-Year PUMS has a record of 7,300,520 households (which represents 6,671,272 housing units and 629,248 group quarters units) and 14,988,864 persons. It is a nationally-representative social survey for the United States.


3

You can check the databases of the INSEE (National Institute of Statistics and Economic Studies). If you are a teacher or student you can use the online statistical data for non-commercial use details over here


2

There was a similar (French) question recently and I posted several answers. The difference may be that you are interested in France-data and the other question was for French-language data. I think the most appropriate answer was for collecting Twitter data, which I'll describe here with some additions to make it specific for France. Twitter data can ...


2

Website of the government open data project. http://www.data.gouv.fr/ Everything you want and more.


2

The US National Intelligence Agency (NGA) maintains a geographic feature database for each country in the world. The data is kept in the Geographic Name Server (GNS). The format is somewhat arcane by today's standards. I have converted each country's dataset into a standardized CSV format to make it more friendly to developers. The France dataset contains ~...


2

Try Stanford Large Network Dataset Collection https://snap.stanford.edu/data/egonets-Facebook.html[ Directed graphs twitter is listed undirected and directed graphs. Hope it answer your question.


2

I was going to add this as a comment on Dan Fowler's answer but I had to create a new account to post official in my role as co-founder and CPO at data.world. I'm glad you found that data.world's upload to work for you. We do cap datasets at 500mb of structured data that we recognize (JSON, CSV, TSV, TTL, N-Triples, Sqllite, Excel). The reason for that is ...


2

Here are two sales data sets from Kaggle: Rossmann Store Sales Walmart Recruiting — Store Sales Forecasting You can ask on the forums to get an answer for the dateset licenses from the competition admins.


1

You can find large public datasets in sources including Google Open Data, Kaggle Datasets, Github and Data.gov. If you are specifically looking for large data coverage, the California Open Data Portal- CA.gov is a great resource. It includes government data and a broad collection of topics.


1

Have you looked at AWS Athena yet? It doesn't require any additional storage (besides the existing S3 bucket) and you can query the data (all of it) whenever you want - for example if your query changes or you get new business requirements. It probably won't give you anything like the "streaming" or real-time characteristics that you seem to be leaning ...


1

Have you thought about Google Spreadsheet? You can retrieve the content of any public Google Spreadsheet in your web app using JSON feeds. The sharing permissions of the Google Spreadsheet should be either “Public” or set to “Anyone with link can view” for the app to fetch cells from the Google Spreadsheet without authentication. You will also need to ...


1

For free sales data, check out this database on Quandl: US Census Bureau - Here are the datasets you'll see if you search for "sales" within this database: https://www.quandl.com/data/USCENSUS-U-S-Census-Bureau?keyword=sales (Disclosure: I work for Quandl)


1

Use Unigraph's data streaming API, it will return you thousands of items per second. Here's how to set it up: https://github.com/unigraph/docs/wiki/Streaming For now just run this sample query in your terminal, to see the amount of results you can get per second: curl -X POST -H 'X-Unigraph-API-Key: [YOUR_API_KEY]' -d 'query UK @stream { node(uid: "1291") {...


1

Zillow has a free API that will return their estimate of average home prices. You can get the data down to zipcode and neighborhood level. http://www.zillow.com/howto/api/APIOverview.htm


1

I guess you are looking for data like this https://archive.ics.uci.edu/ml/datasets/Housing


1

If you're interested in brain imaging data, you could check out the Connectome Coordination Facility I downloaded the original HCP 1200 for our institution a couple of years free of charge. That's 7T & 3T imaging data for 1200 individuals. It's a large dataset but it is freely available for research.


1

Big Data. I think the point has been missed about "Big Data". It doesn't need to be large in volume. There are historically (from 2001) three tenets of big data high volume (e.g. Facebook, Google index) high velocity (e.g. mouse click data being analysed every second for user behaviour triggers) high variety (e.g. data that has a diversity of data types ...


Only top voted, non community-wiki answers of a minimum length are eligible