Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by sargun | Hacker News Reader

Parent Full thread

sargun·

The total toolkit they have is called the Simian Army -- https://github.com/Netflix/SimianArmy

They wrote a blog post about it here: http://techblog.netflix.com/2011/07/netflix-simian-army.html

I don't think they have one that introduces network partitions, but inside a datacenter, network partitions are rare.

View on HN

Inside a single data center they're rare, but when running on AWS in multiple availability zones and regions they're a fact of life which you should be designing your systems to deal with. Given that it would be great to have automated tooling to simulate them and keep everyone on their toes.

Ever lost a switch? Rare, but happens on a regular enough case to prepare against it. Even if you use bonding/teaming, with dual power supply network kit + servers, it can still happen.

> ...but inside a datacenter, network partitions are rare

It's ... complicated.

http://aphyr.com/posts/288-the-network-is-reliable

Yeah, there was an excellent ACM article with him, and Bailis. I think if the network starts to partition, or fail in a datacenter, that's time to evacuate the datacenter / AZ. If a handful of machines fail, they should disengage. If more than say, 5% of the machines in the DC are having reachability issues at any given point (in a modern DC that's like ~2000 machines), it's time to shut it down.

I've seen them happen with the systems I work on a few times over the past five years. It isn't something you should ignore.

Reply on news.ycombinator.com