Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by jshen | Hacker News Reader

Full thread

jshen·

I wonder if their chaos system causes network partitions as well as node failures.

View on HN

The total toolkit they have is called the Simian Army -- https://github.com/Netflix/SimianArmy

They wrote a blog post about it here: http://techblog.netflix.com/2011/07/netflix-simian-army.html

I don't think they have one that introduces network partitions, but inside a datacenter, network partitions are rare.

Inside a single data center they're rare, but when running on AWS in multiple availability zones and regions they're a fact of life which you should be designing your systems to deal with. Given that it would be great to have automated tooling to simulate them and keep everyone on their toes.

Ever lost a switch? Rare, but happens on a regular enough case to prepare against it. Even if you use bonding/teaming, with dual power supply network kit + servers, it can still happen.

I've seen them happen with the systems I work on a few times over the past five years. It isn't something you should ignore.

> ...but inside a datacenter, network partitions are rare

It's ... complicated.

http://aphyr.com/posts/288-the-network-is-reliable

Yeah, there was an excellent ACM article with him, and Bailis. I think if the network starts to partition, or fail in a datacenter, that's time to evacuate the datacenter / AZ. If a handful of machines fail, they should disengage. If more than say, 5% of the machines in the DC are having reachability issues at any given point (in a modern DC that's like ~2000 machines), it's time to shut it down.

Not AFAIK - last time I looked it requires one or more AWS autoscaling groups, and works by semi-randomly terminating instances that match configured criteria.

Reply on news.ycombinator.com