1

Genomic data size has reached volumes to the point, some NCBI has stopped taking certain types of data from Investigators. Other openData projects are also running into barriers..

I know there are tonnes of open source Bittorrent projects: https://github.com/search?o=desc&q=topic%3Abittorrent&s=stars&type=Repositories

What I am wondering is if there is any upperlimit on the file storage that bittorent can handle efficiently?

I know that there are issues regarding seeding, i.e. a lot of users will leech files and stop seeding. Which is probably a major roadblock to adoption of this technology. Has this issue been solved? Is there a way to enforce a certain ratio of upload to download?

Are there any other technologies other than bittorrent that I should look into?

Improve this question

3 Answers 3

Reset to default
1

Is there any upperlimit on the file storage that bittorent can handle efficiently?

No, at least theoretically, there is no upper limit to the size of the file as long as it's supported by the filesystems of the peers.

Has this issue been solved? Is there a way to enforce a certain ratio of upload to download?

Not the public P2P networks, but there are private P2P networks that can maintain the ratio and ensure that the torrent stays 'alive' and supports a good download speed. What you're looking for is called Sharing Ratio Enforcement (SRE).

Are there any other technologies other than bittorrent that I should look into?

If it's an option for you, try setting up your own NAS. And then (if you really want to integrate it within a P2P network, this will be an always active seeder.)

Improve this answer
0
1

Are there any other technologies other than bittorrent that I should look into?

Dat Project was created specifically for these issues. "Dat works on a distributed network unlike cloud services, such as Dropbox or Google Drive. This means Dat transfers files peer to peer, skipping centralized servers. Dat's network makes file transfers faster and more secure. You can even use Dat on local networks for offline file sharing or local backups. Dat reduces bandwidth costs on popular files, as downloads are distributed across all available computers, rather than centralized on a single host." Read more about Dat here.
You're probably going to want to check out another tool this team has created, Beaker Browser, which "is a peer-to-peer browser with tools to create and host websites."

IPFS maybe another option; it bills itself as "the Distributed Web" and goes on to say its "a peer-to-peer hypermedia protocol to make the web faster, safer, and more open."
I haven't even looked beyond the about page, so I can't speak much about this.

Improve this answer
2
  • 1
    Thanks.. this is awesome. I was reading about swarm and ipfs.. didnt know of this.. are you guys planning to/already integrate with incentivizing systems such as ethereum/filecoin etc?
    – alpha_989
    Commented Jul 11, 2017 at 17:34
  • lol. i am not on the dat or beaker teams. just big fan of what they are doing.
    – albert
    Commented Jul 11, 2017 at 18:06
0

Have a look at iRODS, which is in use by many genomics related institutes, e.g. Sanger.

They provide specific protocols for large scale transfer of datasets.

Another noteworthy implementation is HDFS cluster to cluster copy, check distcp.

Also check how certain data is available, e.g. 100.000 genomes do not allow download, but you have to process, where the data lives.

This is also recommended way of doing things Big Data: move the code, not the data.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.