Community wiki to collect data sets available to download/seed on bit torrent
DNS Census, the DNS registration dataset snapshot taken in 2013 (compressed ~15GB and uncompressed 157GB).
The DNS Census 2013 is an attempt to provide a public dataset of registered domains and DNS records. It was inspired by the Internet Census 2012 which showed that ...
BitTorrent is not a great solution for this. Because each file distributed would need its own network of seeds and peers, you'd effectively dilute the network pool with each file you release, leaving you where you started: you being the one doing most of the distribution for most of the files in the first place.
It's probably better to emulate govtrack.us' ...
A few possibilities for using torrents
Offer your torrents through an RSS feed. RSS can be coupled with differential files to have both a distributed network (which will reduce the bandwidth requirements on your end and potentially increase the speed at which your users can download) and reduce the amount of data per download (which will allow end users to ...
Consider whether posting your .torrent file to a BitTorrent index site is the best solution for you. If your objective is to publicise your dataset you may be better off simply posting your .torrent file to a website or forum that focuses on open data or, more specifically, the topic to which the data relates.
You should also bear in mind the large amount ...
From a one time study, I had to decide between multicast transmission or bittorrent on similar specs (10k files/day, 1 to 2GB each plus ~2M files/day, 1k to 2M each). Both technologies try to "swarm up" a bunch of uncooperative individuals into a robust supplier of information.
Anyway, the bittorrent side of the study reads:
Determine the piece size: ...
After Katrina in 2005, the geo community used torrents to distribute imagery data. A site called geotorrents.org was subsequently set up bit was eventually shuttered. A nice write up about torrenting geo data was written up: http://skipperkongen.dk/2011/03/15/bittorrent-and-geodata-was-big-in-2005/
Star Wars Kid Meme data dump - data is a subset of Apache server logs (2003-04-10--2003-11-26) covering the post and subsequent posts that went "viral" across the web. at 1.6gb uncompressed, the author decided to torrent it:
Star Wars Kid: The Data Dump
waxy.org torrent (404s)
waxy.org torrent via Wayback Machine
Several linux distributions chose legittorrents as their primary means of distribution. However, according to the FAQ the site administrators might delist your torrent if it wasn't seeded well (whatever they mean by that).
Are there any other technologies other than bittorrent that I should look into?
Dat Project was created specifically for these issues. "Dat works on a distributed network unlike cloud services, such as Dropbox or Google Drive. This means Dat transfers files peer to peer, skipping centralized servers. Dat's network makes file transfers faster and more ...
Is there any upperlimit on the file storage that bittorent can handle efficiently?
No, at least theoretically, there is no upper limit to the size of the file as long as it's supported by the filesystems of the peers.
Has this issue been solved? Is there a way to enforce a certain ratio of upload to download?
Not the public P2P networks, but there are ...
Step 1: Write a scraper to build a repository of .torrent files or magnet links. You can also use the magnet-hashes from TPB. Or convert a .torrent file to a magnet-link.
Step 2: Pass those magnet-hashes to a script that retrieves the seeder/leecher stats. An example is here.
Step 3: Repeat step 2, as many times and as often as possible.
Step 4: ...