Working with the Chaos Monkey

I think we're going to be seeing a lot more of Chaos Monkey.

CM is a form of active TDD at the system architecture level. This might evolve into setting up partition tests as a prerequisite to instantiating the deployment model (Translation: before you start putting something on a cloud instance, write code that turns the instance off and on from time to time) This assures that the requirements for survival are baked into the app and not something tacked on later after some public failure like the Amazonocolapse.

I was reading on HN the other day a guy talking about Google. He said he saw engineers pull the wires from dozens of routers handling GBs of data -- all without a hitch. The architecture was baked enough that failure was expected.

Many times failure modes like this are burned into hardware, but that kind of design is a long, long, long way from most people's systems.

Netflix is turning out to be my favourite tech company. Just a week ago, in an extensive interview they mentioned that to provide a consistent interface across so many platforms we ended up porting our own version of webkit. And now the Chaos Monkey. It's amazing how technically sound they are considering they were just an online DVD rental company at the start.

The Guiness World record for most steps in a Rube Goldberg device was just set at a competition at Purdue University. The device has 244 steps to water a flower! Now, if you saw the Mythbusters' Christmas episode with the Rube Goldberg device, you know it's really hard to make all those steps go right. But in this one, the engineers used a "hammer test": at any point during the operation of the machine, an engineer could tap the side with a hemmer. If it screwed up, that stage was redesigned. http://www.popularmechanics.com/technology/engineering/gonzo... The end result was the most complex machine of its kind, but it runs very reliably.

When I first read about the Chaos Monkey, I had assumed it was used on their development/staging environment, but this article implies it is on their production system. Does anyone know which is correct?

The Chaos Monkey reminds me of some papers I've read about "crash-only software" and "recovery-oriented computing". With this approach, server software is written assuming the only way it would shutdown is a crash, even for scheduled maintenance. The software must be designed to recover safely every time the service is started. Instead of exercising recovery code paths rarely, they are tested every day.

http://www.armandofox.com/geek/past-projects/recovery-orient...

http://www.usenix.org/events/hotos03/tech/candea.html

Here is an erlang version of the chaos monkey:

    potential_victim(Minions) ->
        fun (Pid) ->
	        not(pman_process:is_system_process(Pid)) 
	    	    and not lists:member(Pid, Minions)
        end.

    death_from_above(Minions) ->
        Pids = lists:filter(potential_victim(Minions), erlang:processes()),
        case Pids of
	    [] -> none;
	    _ ->
	        Victim = lists:nth(random:uniform(length(Pids)), Pids),
	        Name = pman_process:pinfo(Pid, registered_name),
	        exit(Victim, kill),
	        {ok, Victim, Name}
        end.

The idea is to run it during load tests. Afterwards run your normal unit tests to check that nothing got permanently broken. It's good for finding broken supervisor trees.

Akin's Laws of Spacecraft Design [1], law no. 2:

To design a spacecraft right takes an infinite amount of effort. This is why it's a good idea to design them to operate when some things are wrong.

[1] http://spacecraft.ssl.umd.edu/old_site/academics/akins_laws....

Wow cool idea, but I don't think I'd be able to convince my company to do this.

argh! so why was the server crashing? you can't leave me in such suspense....!

Building things this way strikes me as expensive. At Netflix's scale, it pays off, but for systems that don't serve as many requests I'm forced to wonder whether just avoiding the cloud might be more cost-effective.

Even linux has a chaos monkey of sorts. http://linux-mm.org/OOM_Killer

A decade or so ago, I heard computer programming described as a very good occupation for a person who had Asperger syndrome or perhaps limited social skills. I also recall reading then that some surveys of programmers working in that era suggested that those programmers were much more introverted than the general population. But I used to notice when I installed new programs on my Microsoft Windows computer, even after installing Windows 95, that sometimes installing one program would disable another program. That made me wonder if maybe social skills are an essential element of good programming skills. Now when software may have to run in the cloud, interacting with other software hosted on other hardware, all attempting to operate synchronously, wouldn't "software social skills" be rather important for any developer to understand?

> And that's why, even though it sounds crazy, the best way to avoid failure is to fail constantly.

This is my biggest concern with things like large nation-states, large banks, large reinsurance companies, large RAIDs, and large nuclear plants: we centralize resources into a larger resource pool in order to reduce the chances of failure, but in doing so we make the eventual failure more severe, and we reduce our experience in coping with it and our ability to estimate its probability. In fact, we may not even be reducing the chances of failure; we may just be fooling ourselves.

Consider the problem of replicating files around a network of servers. Perhaps you have a billion files and 200 single-disk servers with an MTBF of 10 years, and it takes you three days to replace a failed server.

One approach you can use is to pair up the servers into 100 mirrored pairs and put 10 million files on each pair. Now, about 20 servers will fail every year, leaving ten million files un-backed-up for three days. But the chance that the remaining server of that pair will fail during that time is 3/3650 = 0.08%. That will happen about once every 60 years, and so the expected lifetime of the average file on your system is about 6000 years.

So it's likely that your system will hum along for decades without any problems, giving you an enormous sense of confidence in its reliability. But if you divide the files that will be lost once every 60 years (ten million) by the 60 years, you get about 170 thousand files lost per year. The system is fooling you into thinking it's reliable.

Suppose, instead, that you replicate each file onto two servers, but those servers are chosen at random. (Without replacement.) When a server fails (remember, 20 times a year), there's about a one in six chance that another server will fail in the three days before it's replaced. When that happens, every three or four months, a random number of files will be lost --- about 10 million / 200, or about fifty thousand files, for a total data loss of about 170 thousand files a year. You will likely see this as a major problem, and you will undertake efforts to fix it, perhaps by storing each file on three or four servers instead of two.

This is despite the fact that this system loses data at the same average rate as the other one. In effect, instead of having 100 server pairs to store files on, you have 19,900 partition pairs, each partition consisting of 0.5% of a server. By making the independently failing unit much smaller, you've dramatically increased your visibility into its failure rate, and given yourself a lot of experience with coping with its failures.

In this case, more or less by hypothesis, the failure rate is independent of the scale of the thing. That isn't generally the case. If we had a lot of half-megawatt nuclear reactors scattered around the landscape instead of a handful of ten-gigawatt reactors, it's likely that each reactor would receive a lot less human attention to keep it in good repair. When it threatened to melt down, there wouldn't be a team of 200 experienced guys onsite to fight the problem. There would be a lot more shipments of fuel, and therefore a lot more opportunities for shipments of fuel rods to crash or be hijacked. And so on.

But we might still be better off that way, because instead of having to extrapolate nuclear-reactor safety from a total of three meltdowns of production reactors --- TMI, Tchernobyl, and Fukushima --- we'd have dozens, if not hundreds, of smaller accidents. And so we'd know which design elements were most likely to fail in practice, and how to do evacuation and decontamination most effectively. Instead of Tchernobyl having produced a huge cloud of radioactive smoke that killed thousands or tens of thousands of people, perhaps it would have killed 27, like the reactor failure in K-19.

With respect to nation-states, the issue is that strong nation-states are very effective at reducing the peacetime homicide rate, which gives them the appearance of substantially improving safety. Many citizens of strong nation-states in Europe have never lived through a war in their country, leading them to think of deaths by violence as a highly unusual phenomenon. But strong nation-states also create much bigger and more destructive wars. It is not clear that the citizens of, say, Germany are at less risk of death by violence than the citizens of much weaker states such as Micronesia or Brazil, where murder rates are higher.

I would love to be a developer on a Chaos Monkey.

"Oh no, CM is down, everything's working!"

The blog post seems to imply Stack Exchange is working with the Chaos Monkey when it really isn't. They didn't really build a system that randomly shuts down servers or services. The difference is subtle but important.