Hacker News Reader

Comment by iagooar | Hacker News Reader

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that".

While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.

View on HN

Developers cause software downtime, but they usually don't cause infrastructure downtime, which is what Chaos Monkey does. Cloud VMs have failure rates somewhere around 1-2% depending on who you ask. This is low enough that you can ignore it most of the time, but it'll come back and bite you hard later. Chaos Monkey artificially forces that failure rate high enough that you'll notice problems immediately and fix them before they become too engrained into your architecture.

We have different tools for different kinds of failures.

Chaos Monkey helps ensure that you are resilient to single instance failure. Kong helps ensure that we are resilient to region failure.

Most of the developer-induced pain (which is most frequent source of pain) happens at the service level -- a bad code push that somehow made it through canary, accidentally doing something you shouldn't, misconfiguring something, etc. For tolerating service-level failures, we use different tools that minimize the fallout of the failure injection. Specifically, FIT (and the soon to be revealed ChAP.) These tools allow us to be more surgical in our injection of failure and tie that into our telemetry solutions.

We only inject failures we expect to be resilient to. Sadly, that is a subset of the failures that people cause ;)

The difference is scale. Only a handful of companies run at Netflix scale.

I wouldn't trust developers to do what Chaos Monkey does at such a scale, no matter how good you think they are.

The joke is funny but it is actually shockingly difficult to make nodes kill themselves instead of doing something for more malignant which is the typical case for bugs. That is a zombie just like in real life can be worse than a dead corpse (since we are being funny and all). Chaos Monkey shoots two in the head.

I have had issues killing errant JVMs and Rackspace nodes (yes sadly we are still on Rackspace).

I can understand why 2.0 is much more focused given the plethora of monitoring solutions.

I don't think even hiring top-notch devs would make this reality completely go away. Tools like this would likely put them in check too. And yeah same response on our internal chat here basically x)

From my experience, this is naive. It's funny, but really naive.

The bigger your application stack is (micro-services, API calls, network calls), the more failure you would need to test out and there's no way to "trust" developers to do it themselves.

Also, Netflix hires a lot of top-notch developers and their infrastructure is pretty awesome.

[I don't work @ Netflix, just a devops dude :)]

That rings so true. But on the flip side, it might be that because Netflix has a chaos monkey and all it's services need to be resilient to failures, accidental downtime isn't that big of an issue to them.

Their developers can still make the same mistakes that we do, but their architecture is better designed to handle that. Just a thought.

Netflix had a worldwide outage few weeks ago that last hours so yes they need those kind of tools :)

Reply on news.ycombinator.com