Murder: Fast datacenter code deploys using BitTorrent | Hacker News Reader

Murder: Fast datacenter code deploys using BitTorrent

engineering.twitter.com

63 pointsabraham26 comments

Sometimes I wish geeks wouldn't worry about giving something a cool sounding name and instead just picked something conventional or unremarkable.

I don't want to search Google for help with Murder. I don't want to discuss Murder at the office.

or the next morning....did you complete the murder job?

or commit murder (on github obviously)

or find out who should be blamed for the recent Murder failure.

Don't worry. Murder only takes a few minutes.

The user manual can be called Murder, She Wrote.

At least it's not Mrdr.

One does not simply Bittorrent into Mrdr.

I wonder if they tried multicast. It would be strictly more efficient in terms of bandwidth (one copy per router/hop rather than all the different pieces flying back and forth) and likely less CPU. There's premade software like uftp (http://www.tcnj.edu/~bush/uftp.html) to do this.

Amen. The NIH syndrome at twitter seems to be going strong. They just don't stop reinventing stuff, and usually poorly.

In this case uftp is the time-tested solution, made precisely for this purpose (mass deployment) and provably better on all metrics (speed, bandwidth, security via SSL, complexity).

The mind remains boggled...

You do realize that twitter didn't invent bittorrent, right?

;)

They appear to have invested substantial time into integrating it with their deployment system and improving its performance for their particular network situation rather than using off-the-shelf software like uftp.

You guys know a group of crows is called a Murder, right?

I'd hate to work somewhere where I could only use stuffy, politically correct software names. Call me crazy, but quirky software names are half the reason I release a project.

Indeed. I called my asset manager 'assman'. :)

Context: http://www.youtube.com/watch?v=tujqM2u-BVo

It's a neat idea to use collective nouns at a bird-themed company, there's only one problem: crows aren't known for their tweeting. Crows, well, crow.

Flight, flock, bevy would be pretty generic, pick a songbird and you'd get something more interesting.

As long as it doesn't end in -ly or -ing, I'm pretty much happy.

I remember the uproar when I released a plugin called Shemail.

Couldn't they use a tree-like distribution system where each node only receives pulls from a small number of child nodes?

You could organize this by giving each server an incrementing number and using math to deduce their parent node, much like an implicit binary tree stored as an array: http://en.wikipedia.org/wiki/Binary_tree#Ahnentafel_list

That way you can still use the http or git protocol without having to invent a special flavor of BitTorrent.

He covered why they didn't want to use a tree like distribution method in his presentation.

It introduces a possible single point of failure for all nodes which are children of a failed node.

Maybe this is too easy, but if you're using Git, why not have servers that get a new release be added to a list of servers that can distribute a new release?

Added complexity to continue to scale, and one deploy replica high up in the chain that breaks will break everything downstream. As you start building a system to address those issues, you're adding more complexity. This would lead me toward a Murder based solution from the start.

  no NAT/Firewall issues, no ISP traffic shaping, only trusted peers

Sounds like an easy-to-exploit datacenter to me.

They'd have those protecting the perimenter, but this is only for deploys in a single datacenter right now. They're only doing it on their network, so they don't have to worry about all the hoops torrents normally go through; they can drop the timeouts to nothing, they don't need encryption, etc.

The video states that they will be looking into how best to do deploys across centers in the future, but they are not going to have peers connect across their wan links. One tracker and seeder per location is a first musing.

[Edit: paragraphs]

>The video states that they will be looking into how best to do deploys across centers in the future, but they are not going to have peers connect across their wan links. One tracker and seeder per location is a first musing.

Heck, use a VPN to accomplish this if you're lazy.

css text-shadow on body content is over-rated and actually makes it harder to read.

Reply on news.ycombinator.com