The Discipline of Chaos Engineering

Chaos Engineering seems most useful in the context of distributed systems though, and I'm not entirely certain how you would implement it outside of that context.

I would consider applying invalid inputs, etc. to non-distributed systems to be more along the lines of traditional testing. Perhaps you could implement chaos engineering principles in a non-distributed system by simulating the failure of a CPU core or a region of memory? It seems less useful though, as those things seem very difficult to effectively recover from.

How would you define "chaos engineering" to apply to non-distributed systems?

I agree with Arcsech in that Chaos Engineering is most useful in the context of distributed systems. However, this doesn't mean that you can't learn something new from experimenting in, say, a local VM.

As a matter of fact, Netflix is running a big distributed system, so that's where they focus their testing efforts. In general, I think it's fair to talk about Chaos Engineering and systems in the general sense, distributed or not.

I'd be very surprised if you could get a 1-out-of-2 system with only double the cost. You need twice the hardware and significantly more complex logic.

It depends of what you are trying to do.

About the database : Cache. Offer a different flow that does not need the database, like with hardcoded stuff. Etc etc

About doubling the costs : It highly depends. On a small app, sure it may double the cost. On a big thing with thousands of servers, you can play a bit more with redistributing roles and all.

you're increasing costs, but you're insuring against the lost profits from your site being down as well as reduced consumer trust. there are ways to keep the cost multiplier down (running many smaller instances vs a single large instance, using containers and running mulitenant machines), but when you start thinking about all layers you want to add redundancy (load balancing, application, database, storage, etc.) you're going to be spending quite a bit of additional money for that insurance. which is why there are good arguments that not every application needs to build in these kinds of redundancy.

For your specific question about databases, you generally have clustering to reduce the impact of any one database instance going down, caching of data to guard against temporary db outages/network issues, and sharding of data across multiple databases to reduce the blast radius of any one logical database going away entirely.

1. Netflix has been doing "chaos engineering" for years (earliest public reference I can find is 2011), especially w.r.t. to public cloud.

2. Netflix has done a great job at publicizing their efforts and open sourcing software that helps you do this kind of testing in a continuous, automated fashion.

So I think the "basically what google has been doing" comment is reductive.

Good question.

It is true that Google's Disaster Recovery Testing events are also about breaking things on purpose as a means of preparation. However, those events are typically large-scale, company-wide drills targeting not only critical systems but also business processes involving people.

(They even prevent experts from participating to make sure knowledge is spread across the organization. I recommend reading http://queue.acm.org/detail.cfm?id=2371516 for more.)

As dastbe has pointed out, Chaos Engineering is more about experimenting in a continuous, automated (and hopefully safe) way. Compared to DiRT, experiments are typically smaller in scope, involving fewer people, if any.