AWS Fault Injection Simulator

Doesn’t that kind of defeat the point of chaos engineering, that is, making the process unsupervised and triggering failure scenarios that were not yet known?

I always considered the simplest form of chaos engineering to be a process that randomly kills processes on a server, and see what happens.

If, on the other hand, you mean that it’s important that the “chaos” aspects needs and on/off button and that someone needs to be managing that, then I agree. :)

I blame Heroku, for America us-east is the only option so it's a legacy for many to keep creating resources there.

Regional costs are different. east-1 and west-2 are largely the same, but east-2 and anything outside of the US tend to be more expensive.

The proper way to handle this is using multi-region HA/DR.

For some reason our customers keep asking us to launch our services in us-east-1, which I can only assume is for legacy reasons.

What I don’t understand, however, is what’s preventing Amazon from upgrading the infrastructure in us-east-1 and making it more stable. Is it about mitigating risk of unknown side effects and/or incompatibilities?

The obvious one is if you're physically located near us-east-1 and want low latency to your office. us-east-2 is all the way in Ohio.

Chaos Runner, or anything else that you run on your EC2 instances (or in your ECS containers) requires you to install some kind of agent or daemon. This is non-optimal as it is work in and of itself, it is also hard to correctly simulate things like the network, or various AWS services being down/unreachable.

If you run the agent/daemon on your production stack, then it's a potential vector for misconfiguration or attack. But if you don't run the agent/daemon in production, then it's another way in which your test stack diverges from production!

I saw various PR/FAQs related to Chaos engineering while I worked in both EC2 and the AWS developer tools org. I've been gone over a year now, but I would bet that FIS does something at the EC2 Network level so that you don't have to install stuff on your instances or containers.

Inattention-to-detail as a service? I love it!

Hire me to do a half-assed job so your infrastructure can randomly fail to make it more resilient.

> Me neither. I would expect AWS to provide this service for free.

A truly cunning pricing scheme would be to strike a Coasian Bargain: "we inject faults for free! If you want us to stop, you'll need to pay extra". That would mean the cost would fall mostly on those who are unable to avoid paying it.

A route 53 propagation issue caused a bunch of our web servers to be unable to connect to the read replicas of our RDS cluster. The cluster scaled in a RR, but the DNS change took hours to propagate. The web servers kept trying to connect to the replica that didn't exist.

Our workaround was to put the IP address in the hosts files - not an ideal setup, but it got the job done.

I know -- still no mention of the state of the art or what product it might replace.

It would be better if that context carried over to this document.

Not saying you're wrong, but maybe Wikipedia is leaving off a lot of relevant history on the Chaos Engineering article then.

Edit: Funny how my original comment is downvoted for contradictory reasons, both because "obviously that was irrelevant to the state of the art in chaos engineering", and because "oh don't worry, that's relevant, they just discuss it in other contexts".

People have been doing Chaos Engineering, and the more general category of "DevOps", for years before those things were given labels. I think we may credit Netflix with the awesome name of Chaos Monkey and for building a systemic in-house practice, but they were not the first to perform intentional destructive resilience testing in production.

Case in point, I've been yanking out active drives, cables, PSUs and CPUs since the 1980s, and not always by accident.

What's more, I was taught to do by someone who'd been babysitting VAXes for years; they had a ceramic axe that, at the age of fourteen, I'd seen them swing right through a three-phase minicomputer power feed at my local university's data center "for test purposes".

Chaos Monkey was explained to me as "The (virtual) monkey swinging around the (virtual) data center unplugging the (virtual) cables". It very well could have been based on Robbins. The point wasn't originality - the point was to have the same benefit in our burgeoning cloud environment.

In the 80s we had a serial line in our test lab set up with a chain of paperclips on the TX and RX lines.

Made it easy to insert comms errors and failures.

didn't realize this. and I went to re:invent last year!