I know that there are political issues (my place of work bans all peer-to-peer software without an explicit waiver), but how well does bit torrent scale for distributing large data collections where we may want to be able to pull out specific granules?

I know of groups using BitTorrent to distribute the whole collection as a single download, or a couple dozen items, and even a few thousand.

... but I'm serving one data collection that's adding ~60k files, 16MB each ... per day. Are there servers for hosting BT that peform well when serving millions of files?

If not, how far do they scale vertically before I have to start going horizontally?

  • Were you thinking each day's dump would be its own torrent, or each file individually? May 12, 2013 at 17:07
  • @SideOfBacon : that's why I want to know what the limitations are ... is it practical to list each file independently? How about every past calibration of each file? Or tiles of the file so that someone can then selectively retrieve parts? ... what are the practical min/max file sizes for an individual torrent, and how many torrents can you seed from single server (or how do you calculate the max based on CPU/RAM or other resources?)
    – Joe
    May 12, 2013 at 17:13
  • 1
    That's a lot of questions. A better question might be "based on data with this characteristics (size, frequency of updating, etc.) what is the best way to archive and distribute it?" Otherwise, you may be falling into the XY Problem: meta.stackexchange.com/questions/66377/what-is-the-xy-problem May 12, 2013 at 19:40
  • @SideOfBacon : But that's not my question. I'm attempting to figure out what the technical limitations of BitTorrent are to determine if it's even a candidate for me to consider testing. PS.. I know what the XY Problem is ... it's been around since before StackOverflow.
    – Joe
    May 13, 2013 at 1:11

4 Answers 4


BitTorrent is not a great solution for this. Because each file distributed would need its own network of seeds and peers, you'd effectively dilute the network pool with each file you release, leaving you where you started: you being the one doing most of the distribution for most of the files in the first place.

It's probably better to emulate govtrack.us' rsync strategy. Perhaps also with terms that say that anybody rsyncing must run an open rsync themselves (or voluntarily, depending on the size and happiness of your community) This will cut down on redundant downloading, and give you some bandwidth advantages.

Here's how govtrack does it: http://www.govtrack.us/developers/rsync

  • I'm actually okay with being the primary seed -- it's when some major event happens (eg, X-class flares, CMEs), we get a sudden burst of traffic ... I'm hoping to be able to even out some of the burstiness, and allow someone to more quickly get their data if someone else locally already has it. My boss has referred to our data as 'write one, read never' as the majority of the data will never be downloaded except for automated processes.
    – Joe
    May 13, 2013 at 1:30
  • Oh ... and the data volume is big enough that you actually can't keep up with it via rsync over the open internet ... we run about 30 parallel rsyncs to emulate some of the behaviour that bit torrent does natively with using multiple streams. Luckily, there's only 4 sites that pull all of the data as soon as it's available, and they act as caches for distributing to everyone else ... but in part the question is to determine feasability in using BT to spread the load across the caches rather than just using the geographically closest one.
    – Joe
    May 13, 2013 at 1:37
  • Oh .. and I'm interested in your comment about 'dilute the network pool' ... do you have any info on when that starts to happen?
    – Joe
    May 13, 2013 at 1:38

A few possibilities for using torrents

  1. Offer your torrents through an RSS feed. RSS can be coupled with differential files to have both a distributed network (which will reduce the bandwidth requirements on your end and potentially increase the speed at which your users can download) and reduce the amount of data per download (which will allow end users to get the data faster).
  2. Use a technology like bittorrent's Sync
  • I'm not asking how to find the torrents -- I have a search engine for that ... I just want to know what the limits are if I tried to seed millions of files. The differentials are interesting, but problematic as many people only download one image a minute, or at some reduced cadence .... then download the full set once they identify a time period of interest.
    – Joe
    May 13, 2013 at 1:42
  • @Joe, I'm confused. I didn't say anything about a search engine... if you are referring to the RSS that is simply a way for the bittorrent client to download things autonomously and keep a dataset updated... I hope this information helps.
    – John
    May 13, 2013 at 2:20
  • John, I assumed the RSS feed would be to give people the .torrent files so they can then download the files ... but we have other mechanisms (a search engine) for me to get them that part ... it's the actual seeding & BT protocol that I need to get up to speed on. If I didn't understand what you were suggesting with the RSS feed, and it wasn't for distributing the .torrent files or magnet links, could you please provide more detail to clarify?
    – Joe
    May 13, 2013 at 12:08

From a one time study, I had to decide between multicast transmission or bittorrent on similar specs (10k files/day, 1 to 2GB each plus ~2M files/day, 1k to 2M each). Both technologies try to "swarm up" a bunch of uncooperative individuals into a robust supplier of information.

Anyway, the bittorrent side of the study reads:

  1. Determine the piece size: Typically between 256 kB and 1 MB but size can be chosen freely. They are atomic in the sense that a typical client will download a given piece from one peer only. If that peer fails to deliver the whole piece within a given time range, no integrity checks can be done, the data has to be discarded and rerequested off another peer in the swarm.

    A piece is integrity-protected by a 20B sha1 sum. So for a piece size of 256k and your payload data size of 960G, there will be roughly 3.75M sha1s in your torrent file, totalling to 75 MB.

    Had you chosen, say 4 MB, for a piece, your torrent would "only" be 4.4 MB in size but now your swarm must be more cooperative, ideally it should consist of 240k individuals each willing to serve 4 MB in one go, realistically you'd probably see something like 1k peers.

  2. Torrent structure: You can have arbitrary file/directory layouts in your torrent (each adding their name to the total size of the torrent). Short file names with a flat hierarchy will keep the torrent size down but will impede usability on the consumer's side.

    By that I mean, if you want people to continue hosting the final download result in the consumer's file system must be usable directly. Most bittorent clients will keep seeding off that directory and they're bad at tracking file renamings, moving, etc. (and I'm willing to bet that most users won't symlink 60k files off your structure into their own structure).

    Also, I found for the very same reasons you must serve raw data, i.e. something a typical tool in the domain can process directly. Delivering 60k compressed files (whilst saving a lot of bandwidth) will highly affect user-experience and swarm cooperativeness. The user will end up with 2 copies of that giant pile data, the compressed one for seeding and an uncompressed one for further use.

    Note that many torrent clients allow to selective download only a few files in a big torrent file, so users that only need 4 out of your 60k files might happily download and seed those very 4 files as long as they don't have to extract them from a much larger compressed archive of files.

  3. Incentivise: This is the most debatable point and I'm keen on finding more profound studies on this. There are some for the multicast case though so I will just draw some analogies here. The idea is that completely unmonetised swarms have no reason not to be selfish (i.e. they will just leech what they need off the swarm and move on).

    In theory it greatly helps to introduce some kind of currency that seeders will receive and leechers will have to pay. In practice it's hard to find the "right" currency. Multicast ecosystems use trust as a payable and better latency as a receivable. Bittorrent ecosystems might use something like a ratio watch, though I'm not sure how to convert that "ratio currency" into something useful.

Anyway, for my scenario bittorrent did not turn out practical, even though we ticked off 1. and 2. of my list.

Edit: Oh and re your vertical/horizontal issue: I found that with a carefully crafted stack (ext4, no readdir() calls, getdents() with big buffers) you can easily go up to 3M files per directory, but we recently migrated to max 32768 files per dir and prefixed dirs on the level up. However, with bittorrent you can't really control what your consumers use.

  • Thanks ... I didn't even think about the problems that might arise on the client side -- lack of hashed directories mean that lots of files will get messy really quickly. You almost need to write your own client to deal w/ that and file renaming / reorganization. (it's currently bad enough without bittorrent that there's a GSoC project to write something to help scientists search their local data)
    – Joe
    Jun 27, 2013 at 11:14

After Katrina in 2005, the geo community used torrents to distribute imagery data. A site called geotorrents.org was subsequently set up bit was eventually shuttered. A nice write up about torrenting geo data was written up: http://skipperkongen.dk/2011/03/15/bittorrent-and-geodata-was-big-in-2005/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.