Netflix Chaos Monkey Upgraded

I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.

Was wondering the same thing... I know in our environment, unexpected events on a box cause more problems than entire server failures. We design around servers coming in and out; specific processes failing in random ways is harder to design around.

If you can handle a server failing and you have good reporting, you can handle many of those random issues by simply rebooting the affected server. So I can see them taking out things like "load up the memory" or "load up the CPU". Logic errors (bad RAM, corrupted packets, high packet loss) are another story, but I don't know if V1 did those.

See my other reply in this thread for our reasoning. https://news.ycombinator.com/item?id=12744567

I think it could be because it doesn't provide the value to the development teams. Why chase a memory leak or a CPU bug if it's just being caused by your fault testing app?

To prevent the negative effect of random machines dissapearing though.. that's a challenge that involves good ops, devs, even UI/UX I would imagine, and closer to something that users experience negatives because of in real life.

To test a distributed system, there's not a lot of value to simulate all the many different conditions that can happen on a machine. From the system's perspective, you don't really care what happens on a single computer.

The conditions you need to simulate are (a) the machine being abruptly gone or (b) the machine still accepting requests, but being very slow in returning them and maybe (c) machine returning incorrect results. Seen from the outside, everything that can happen is usually (a) or (b), and with ECC memory and reasonable software (?) hopefully never (c).

Netflix had a worldwide outage few weeks ago that last hours so yes they need those kind of tools :)

The joke is funny but it is actually shockingly difficult to make nodes kill themselves instead of doing something for more malignant which is the typical case for bugs. That is a zombie just like in real life can be worse than a dead corpse (since we are being funny and all). Chaos Monkey shoots two in the head.

I have had issues killing errant JVMs and Rackspace nodes (yes sadly we are still on Rackspace).

I can understand why 2.0 is much more focused given the plethora of monitoring solutions.

That rings so true. But on the flip side, it might be that because Netflix has a chaos monkey and all it's services need to be resilient to failures, accidental downtime isn't that big of an issue to them.

Their developers can still make the same mistakes that we do, but their architecture is better designed to handle that. Just a thought.

The difference is scale. Only a handful of companies run at Netflix scale.

I wouldn't trust developers to do what Chaos Monkey does at such a scale, no matter how good you think they are.

I don't think even hiring top-notch devs would make this reality completely go away. Tools like this would likely put them in check too. And yeah same response on our internal chat here basically x)

We have different tools for different kinds of failures.

Chaos Monkey helps ensure that you are resilient to single instance failure. Kong helps ensure that we are resilient to region failure.

Most of the developer-induced pain (which is most frequent source of pain) happens at the service level -- a bad code push that somehow made it through canary, accidentally doing something you shouldn't, misconfiguring something, etc. For tolerating service-level failures, we use different tools that minimize the fallout of the failure injection. Specifically, FIT (and the soon to be revealed ChAP.) These tools allow us to be more surgical in our injection of failure and tie that into our telemetry solutions.

We only inject failures we expect to be resilient to. Sadly, that is a subset of the failures that people cause ;)

Developers cause software downtime, but they usually don't cause infrastructure downtime, which is what Chaos Monkey does. Cloud VMs have failure rates somewhere around 1-2% depending on who you ask. This is low enough that you can ignore it most of the time, but it'll come back and bite you hard later. Chaos Monkey artificially forces that failure rate high enough that you'll notice problems immediately and fix them before they become too engrained into your architecture.

From my experience, this is naive. It's funny, but really naive.

The bigger your application stack is (micro-services, API calls, network calls), the more failure you would need to test out and there's no way to "trust" developers to do it themselves.

Also, Netflix hires a lot of top-notch developers and their infrastructure is pretty awesome.

[I don't work @ Netflix, just a devops dude :)]

I'd love to see this used in the Jepsen tests

What do you use for high availability with Docker?

Blockade is another tool to check out in this area: http://blockade.readthedocs.io/en/latest/

Yes, I wonder that too - complete node failure is the nicest failure that can happen. (See e.g. http://danluu.com/limplock/, because Dan Luu's site is always excellent.)

"We rewrote the service for improved maintainability" seems an important part of this blog post.

Maybe internally such situations translate to a termination anyway.

Resource exhaustion manifests as latency or failure. We inject latency and failure using FIT, so we can limit the "blast radius". When you are testing these failure modes, you are more testing the interaction between micro services, and this requires a bit more precision and sophistication.

Source: I'm on the Chaos team here at Netflix.

perhaps it had too much liability.

I plan to run on it on staging while running load test suites. Not brave enough to run it in production yet

If you don't use a tool like this, entropy will take care of taking your machines down for you. Only then, it won't be a regularly rehearsed part of "normal operations" so you might find yourself up creek without a paddle.

Go intentionally introduces randomness when reading maps so developers don't write code dependent on order.

The Linux kernel also recently added a debugging option that when enabled, instead of just probing a device it performs a probe / remove / probe sequence to ensure that device removal works.

How so? The benefits are worth it, and I doubt any CEO will be argue against having fault tolerant code :)

You catch bugs, and no one says you can't run Chaos Monkey in staging or a similar environment if it really is a tough sell.

Chaos Monkey is sort of like Advanced Continuous Deployment. Most shops are still struggling with the basics. You cant even think of trying to sell this running to the C-level until you've proven that you can at least walk (automated deployment and rollback).

I remember reading years and years ago about bandit algorithms... this kind of ops work is at a level that's found only in a few different companies.

Yes.

The "sell" can be tricky for some people, until your first production issue. Never let a crisis go to waste.

Machines go away all the time in the cloud. This tool increases the frequency so you can ensure your system handles it gracefully.

Some people believe their system can tolerate this class of failures, but without continuous validation, that is more of a hope than a certainty.

Yes, we have.

See https://medium.com/production-ready/chaos-monkey-for-fun-and...

As for chaos being a hard sell: https://medium.com/production-ready/chaos-engineering-a-shif...

In my opinion, it doesn't count unless its in production.

Why? Your customers use your production environment, not your test environment. Something will cause loss of an instance for you: * Mistaken termination * AWS retirement (and you missed the email) * Cable trip in the data center * <Something else we can come up with> * <This list goes on>

So, vaccinate against the loss of an instance cratering your service. Give your prod environment a booster shot (with Chaos Monkey or something like it) every hour of every day. Then, when anything from the above list happens you're infrastructure handles it gracefully and without intervention. Continued booster shots ensure that this stability continues through config changes, software version changes, OS changes, tooling changes, etc.

I think the better question is "Why wouldn't you do this?"

Just wait till it starts killing developers to test your bus factor.

It's up.

down : Secure Connection Failed

The connection to techblog.netflix.com was interrupted while the page was loading.

Every time that URL comes up people try to access it from https but the site is only available from http... Fix your Firefox, it's clearly at fault here.