8

Here you are: A MOOC from coursera, but you have to wait until they will run it again or You can start from dozens of courses here in big data university.


7

I'm sorry to hear you had a frustrating time uploading to datahub.io and that the site wasn't clear enough about the size limit. Another datahub-like platform that we have worked with in the past is data.world. Their size limit is 500MB.


7

BitTorrent is not a great solution for this. Because each file distributed would need its own network of seeds and peers, you'd effectively dilute the network pool with each file you release, leaving you where you started: you being the one doing most of the distribution for most of the files in the first place. It's probably better to emulate govtrack.us' ...


6

A few possibilities for using torrents Offer your torrents through an RSS feed. RSS can be coupled with differential files to have both a distributed network (which will reduce the bandwidth requirements on your end and potentially increase the speed at which your users can download) and reduce the amount of data per download (which will allow end users to ...


5

I also think that Twitter data is the best way to get >100Gb french dataset. OpenStreetMaps gives really great open data but you can't reach 100Gb easily : the Integral Metropolitan France dataset is 2.6 GB. There is no open weather data by MeteoFrance (you have to pay the licence) but there is an Open Meteo Forecast that you can try. I never had the ...


5

Not sure if this answers the question, but there are two aspects: (1) using algorithms to analyze the holdings in a data catalog or a large (big) dataset; and (2) gathering data around genetics and genomics. (1) There are many tools and programs underway on data analytics. Check out a large solicitation from NSF and an interagency big data initiative that ...


5

From a one time study, I had to decide between multicast transmission or bittorrent on similar specs (10k files/day, 1 to 2GB each plus ~2M files/day, 1k to 2M each). Both technologies try to "swarm up" a bunch of uncooperative individuals into a robust supplier of information. Anyway, the bittorrent side of the study reads: Determine the piece size: ...


4

After Katrina in 2005, the geo community used torrents to distribute imagery data. A site called geotorrents.org was subsequently set up bit was eventually shuttered. A nice write up about torrenting geo data was written up: http://skipperkongen.dk/2011/03/15/bittorrent-and-geodata-was-big-in-2005/


4

In addition to educational materials, Cloudera offers a free Hadoop cluster that is ready to go. After installing VirtualBox, KVM or VMWare, you can download and run the virtual machine. link blog post youtube tutorial Hortonworks also offers a complete VM: "sandbox. This comes with Hive, Pig, and other tools that can be managed by the browser frontend. ...


4

They are not the same at all. Datasets are Open if they are available under a free license to everyone. Datasets are Big if... well, they are. Typically big beyond where common software can handle them in real time. For example Facebook and Google work with Big Data that is not Open. Most Open Data sets are actually an example of Small Data: The datasets ...


4

You're probably going to need to get information from many jurisdictions - at least in the US, there is not a good/granular sources for property tax, crime, and other information for all states in one dataset. For crime data, check out FBI UCR reports which are usually not granular. Each city may (hopefully) report their own data. Try google and/or http://...


4

If you want quick, dirty, and free, you can always host downloadable datasets at the Internet Archive (archive.org). No pretty interface, but as much storage as you could ever want.


4

"The Unreasonable Effectiveness of Data" is an insightful article, written by by some Google researchers in 2010. If you search for articles that have cited this, you'll find plenty of stuff to read.


3

Here you can find tons of APIs: https://www.programmableweb.com/category/all/apis You also have a filter for streaming APIs: https://www.programmableweb.com/category/streaming/apis?category=20253


3

The quick, dirty, easy answer: compress the large data files into multiple .7z files, setting the maximum limit to 95mb per compressed file. I do this to get around GitHub's limitations, and want to say I've done it on datahub.io too, though I can't recall exactly. I have done it on CKAN instances, so I guess that covers it. Note: GLFS (Git Large File ...


3

At least a million records you say? I know of a data set that could fit the bill. The 2013 ACS 5-Year PUMS has a record of 7,300,520 households (which represents 6,671,272 housing units and 629,248 group quarters units) and 14,988,864 persons. It is a nationally-representative social survey for the United States.


3

You can check the databases of the INSEE (National Institute of Statistics and Economic Studies). If you are a teacher or student you can use the online statistical data for non-commercial use details over here


2

Website of the government open data project. http://www.data.gouv.fr/ Everything you want and more.


2

The US National Intelligence Agency (NGA) maintains a geographic feature database for each country in the world. The data is kept in the Geographic Name Server (GNS). The format is somewhat arcane by today's standards. I have converted each country's dataset into a standardized CSV format to make it more friendly to developers. The France dataset contains ~...


2

There was a similar (French) question recently and I posted several answers. The difference may be that you are interested in France-data and the other question was for French-language data. I think the most appropriate answer was for collecting Twitter data, which I'll describe here with some additions to make it specific for France. Twitter data can ...


2

Here are two sales data sets from Kaggle: Rossmann Store Sales Walmart Recruiting — Store Sales Forecasting You can ask on the forums to get an answer for the dateset licenses from the competition admins.


2

Try Stanford Large Network Dataset Collection https://snap.stanford.edu/data/egonets-Facebook.html[ Directed graphs twitter is listed undirected and directed graphs. Hope it answer your question.


2

I was going to add this as a comment on Dan Fowler's answer but I had to create a new account to post official in my role as co-founder and CPO at data.world. I'm glad you found that data.world's upload to work for you. We do cap datasets at 500mb of structured data that we recognize (JSON, CSV, TSV, TTL, N-Triples, Sqllite, Excel). The reason for that is ...


1

Have you looked at AWS Athena yet? It doesn't require any additional storage (besides the existing S3 bucket) and you can query the data (all of it) whenever you want - for example if your query changes or you get new business requirements. It probably won't give you anything like the "streaming" or real-time characteristics that you seem to be leaning ...


1

Have you thought about Google Spreadsheet? You can retrieve the content of any public Google Spreadsheet in your web app using JSON feeds. The sharing permissions of the Google Spreadsheet should be either “Public” or set to “Anyone with link can view” for the app to fetch cells from the Google Spreadsheet without authentication. You will also need to ...


1

For free sales data, check out this database on Quandl: US Census Bureau - Here are the datasets you'll see if you search for "sales" within this database: https://www.quandl.com/data/USCENSUS-U-S-Census-Bureau?keyword=sales (Disclosure: I work for Quandl)


1

Use Unigraph's data streaming API, it will return you thousands of items per second. Here's how to set it up: https://github.com/unigraph/docs/wiki/Streaming For now just run this sample query in your terminal, to see the amount of results you can get per second: curl -X POST -H 'X-Unigraph-API-Key: [YOUR_API_KEY]' -d 'query UK @stream { node(uid: "1291") {...


1

Zillow has a free API that will return their estimate of average home prices. You can get the data down to zipcode and neighborhood level. http://www.zillow.com/howto/api/APIOverview.htm


1

I guess you are looking for data like this https://archive.ics.uci.edu/ml/datasets/Housing


1

Big Data. I think the point has been missed about "Big Data". It doesn't need to be large in volume. There are historically (from 2001) three tenets of big data high volume (e.g. Facebook, Google index) high velocity (e.g. mouse click data being analysed every second for user behaviour triggers) high variety (e.g. data that has a diversity of data types ...


Only top voted, non community-wiki answers of a minimum length are eligible