Score:0

What's the fastest way to distribute a large file between a fleet of machines?

dz flag
Wug

Imagine you run a highly parallel service which requires you to sync a large file (let's say 100GB) between many machines (let's say 100) on a regular basis, and you want to get this data distributed as quickly as possible, so the service which processes that file downstream can begin to do its work.

Sequentially sending it to machine one, then machine two, then machine three, etc etc, is easy to script, but very time consuming, and very wasteful: at any given time, 98 machines are sitting idle.

This wikipedia page describes a number of different algorithms for broadcasting data to multiple machines. Are there any shell utilities which use a smarter-than-naive strategy to perform this broadcast operation on a regular file?

ar flag
Which version of Ubuntu are you using? If all the machines are in the same network, setting up bittorrent client-server on all the machine may work best. See [Transmission How To](https://help.ubuntu.com/community/TransmissionHowTo) for more.
qwr avatar
kr flag
qwr
I don't think this is necessarily an Ubuntu question but a networking one
Score:1
aq flag

If they are on the same network, any mutlicast FTP software should work (faster than bittorrent): uftp, udpcast, and flamethrower are available in apt.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.