The cold reality of the Kinesis Incident
cloudirregular.substack.com
cloudirregular.substack.com
This is also wrong:
> That’s why the shipbuilders bragged that the Titanic was unsinkable. Instead, systematic design error let water spill from one compartment to the next. The moment an iceberg compromised one section of the hull, the whole ship was doomed.
The Titanic was built to withstand as many as four flooded sections and still remain afloat. No one imagined a scenario in which more than four sections would flood, but then the iceberg ripped open six sections and down it went. Was this a systemic failure? I think it's more of a failure to anticipate a possible worst case scenario. An argument could be (and has been) made that if the watertight sections had been taller, the flooding would have been slower and there would have been more time for aid to arrive. That was arguably a flaw in the system of bulkheads and thus a systemic failure.
- people using us-east-1 when it has been shown to be the most problematic AWS region
- companies building shitty applications and devices that require the cloud to function and not considering what happens when the cloud/internet is down
- people assuming the cloud can't/won't go down or have issues between the end user and the server
These types of events will continue to happen, so for the love of god stop using us-east-1
This probably isn't the best use of chaos monkey, though I'm not sure. There's probably room in here somewhere for better tools for modeling failure.
Some workloads can't be split, but they are pretty small, especially when you start getting to medium load levels (anything involving sharding.) Mostly then its down to cost.