Ask HN: How to update a large network of computers | Hacker News Reader

Ask HN: How to update a large network of computers

1 pointhuherto9 comments

I am working for a company that has several thousand stores located in a large geographic area. Every time data changes have to be delivered to the stores (around 500MB each) there are about a dozen servers transferring the info to each individual store. As you probably imagine it takes a long time to do it and it requires a lot of bandwidth. It will be faster if the stores could be updated from other stores, specially if they are close from each other. Do you know of any software that can help us do it. May be some type of mirroring or maybe git or some other version control system work for it. Any suggestions?

View on HN

A friend of mine told me this was a Google interview question. Anyways, I'd recommend taking a look at this:

The Frisbee server uses a custom multicast protocol to distribute highly-compressed disk images to consenting clients. The Frisbee client is a multi-threaded marvel that receives, decompresses, and smacks data down on your hard drive, really, really fast. Frisbee can be used to install complete disk images or single partition images.

- http://www.cs.utah.edu/flux/papers/frisbee-usenix03

- http://www.emulab.net/software.php3

http://en.wikipedia.org/wiki/BitTorrent (i would link to the protocol but the link doesn't show up correctly here)

there are probably better options, but this one is already implemented and likely wouldn't be that hard to make work for your situation. might require you writing something yourself to suit your needs, or you might be able to find a turnkey solution.

Ah, thanks I will look into it

Some other thoughts:

You mention data changes. So what about sending just the changes?

What is the real problem? That it takes a lot of time? Or that it requires bandwidth?

If downloads are slow, are your central servers overloaded? What about putting the data on a CDN or Amazon S3?

The time.

Sounds like you simply don't have enough bandwidth then. Fortunately that is an easily solvable problem in 2008 :-)

Assuming these stores are behind DSL ... DSL upload speeds are usually terribly slow. Are you sure that it would actually be faster than downloading from some central server?

I think so, even if the DSL upload is slower, we would not have the bottle neck in the central servers.

Here's another useful tool I just remembered.

http://saf.bio.caltech.edu/nettee.html

Reply on news.ycombinator.com