Hacker News Reader

Comment by andrewguenther | Hacker News Reader

Interesting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differentiating features of Chaos Monkey. Did they just not get a lot of use internally at Netflix?

View on HN

Resource exhaustion manifests as latency or failure. We inject latency and failure using FIT, so we can limit the "blast radius". When you are testing these failure modes, you are more testing the interaction between micro services, and this requires a bit more precision and sophistication.

Source: I'm on the Chaos team here at Netflix.

For reference, FIT: http://techblog.netflix.com/2014/10/fit-failure-injection-te...

New life goal: Join the "Chaos Team"

Yes, I wonder that too - complete node failure is the nicest failure that can happen. (See e.g. http://danluu.com/limplock/, because Dan Luu's site is always excellent.)

"We rewrote the service for improved maintainability" seems an important part of this blog post.

I wonder if they have a service that automatically detects all those failures and terminates the entire node instead?

I was thinking along these lines as well. Possibly they have other checks and balances in place for less than death situations for servers like high CPU usage and those situations result in another mechanism killing the server which is the same as what this solution provides.

We do: http://techblog.netflix.com/2015/07/tracking-down-villains-o...

Maybe internally such situations translate to a termination anyway.

perhaps it had too much liability.

Reply on news.ycombinator.com