What are the practical limits of releasing open data via bit torrent?

Question

I know that there are political issues (my place of work bans all peer-to-peer software without an explicit waiver), but how well does bit torrent scale for distributing large data collections where we may want to be able to pull out specific granules?

I know of groups using BitTorrent to distribute the whole collection as a single download, or a couple dozen items, and even a few thousand.

... but I'm serving one data collection that's adding ~60k files, 16MB each ... per day. Are there servers for hosting BT that peform well when serving millions of files?

If not, how far do they scale vertically before I have to start going horizontally?

Answer 1

BitTorrent is not a great solution for this. Because each file distributed would need its own network of seeds and peers, you'd effectively dilute the network pool with each file you release, leaving you where you started: you being the one doing most of the distribution for most of the files in the first place.

It's probably better to emulate govtrack.us' rsync strategy. Perhaps also with terms that say that anybody rsyncing must run an open rsync themselves (or voluntarily, depending on the size and happiness of your community) This will cut down on redundant downloading, and give you some bandwidth advantages.

Here's how govtrack does it: http://www.govtrack.us/developers/rsync

Answer 2

A few possibilities for using torrents

1. Offer your torrents through an RSS feed. RSS can be coupled with differential files to have both a distributed network (which will reduce the bandwidth requirements on your end and potentially increase the speed at which your users can download) and reduce the amount of data per download (which will allow end users to get the data faster).
2. Use a technology like bittorrent's Sync

Answer 3

After Katrina in 2005, the geo community used torrents to distribute imagery data. A site called geotorrents.org was subsequently set up bit was eventually shuttered. A nice write up about torrenting geo data was written up: http://skipperkongen.dk/2011/03/15/bittorrent-and-geodata-was-big-in-2005/

Answer 4

From a one time study, I had to decide between multicast transmission or bittorrent on similar specs (10k files/day, 1 to 2GB each plus ~2M files/day, 1k to 2M each). Both technologies try to "swarm up" a bunch of uncooperative individuals into a robust supplier of information.

Anyway, the bittorrent side of the study reads:

1. Determine the piece size: Typically between 256 kB and 1 MB but size can be chosen freely. They are atomic in the sense that a typical client will download a given piece from one peer only. If that peer fails to deliver the whole piece within a given time range, no integrity checks can be done, the data has to be discarded and rerequested off another peer in the swarm.

   A piece is integrity-protected by a 20B sha1 sum. So for a piece size of 256k and your payload data size of 960G, there will be roughly 3.75M sha1s in your torrent file, totalling to 75 MB.

   Had you chosen, say 4 MB, for a piece, your torrent would "only" be 4.4 MB in size but now your swarm must be more cooperative, ideally it should consist of 240k individuals each willing to serve 4 MB in one go, realistically you'd probably see something like 1k peers.

2. Torrent structure: You can have arbitrary file/directory layouts in your torrent (each adding their name to the total size of the torrent). Short file names with a flat hierarchy will keep the torrent size down but will impede usability on the consumer's side.

   By that I mean, if you want people to continue hosting the final download result in the consumer's file system must be usable directly. Most bittorent clients will keep seeding off that directory and they're bad at tracking file renamings, moving, etc. (and I'm willing to bet that most users won't symlink 60k files off your structure into their own structure).

   Also, I found for the very same reasons you must serve raw data, i.e. something a typical tool in the domain can process directly. Delivering 60k compressed files (whilst saving a lot of bandwidth) will highly affect user-experience and swarm cooperativeness. The user will end up with 2 copies of that giant pile data, the compressed one for seeding and an uncompressed one for further use.

   Note that many torrent clients allow to selective download only a few files in a big torrent file, so users that only need 4 out of your 60k files might happily download and seed those very 4 files as long as they don't have to extract them from a much larger compressed archive of files.

3. Incentivise: This is the most debatable point and I'm keen on finding more profound studies on this. There are some for the multicast case though so I will just draw some analogies here. The idea is that completely unmonetised swarms have no reason not to be selfish (i.e. they will just leech what they need off the swarm and move on).

   In theory it greatly helps to introduce some kind of currency that seeders will receive and leechers will have to pay. In practice it's hard to find the "right" currency. Multicast ecosystems use trust as a payable and better latency as a receivable. Bittorrent ecosystems might use something like a ratio watch, though I'm not sure how to convert that "ratio currency" into something useful.

Anyway, for my scenario bittorrent did not turn out practical, even though we ticked off 1. and 2. of my list.

Edit: Oh and re your vertical/horizontal issue: I found that with a carefully crafted stack (ext4, no readdir() calls, getdents() with big buffers) you can easily go up to 3M files per directory, but we recently migrated to max 32768 files per dir and prefixed dirs on the level up. However, with bittorrent you can't really control what your consumers use.