Hacker News Reader

It sounds like this can be mitigated by making sure everything is run in dry run mode first, and for something mission critical, getting it double-checked by someone before removing the dry run constraint.

It's good practice in general, and I'm kind of astonished it's not part of the operational procedures in AWS, as this would have quickly been caught and fixed before ever going out to production.

View on HN

I'm not sure you've understood what the problem is. They were removing some servers from a group. This isn't something that get's a dry run, not without spinning up the entire AWS infrastructure. It also wouldn't have helped a jot, since the issue came about after an employee executing a playbook made a typo.

There's no way this could have been mitigated with a dry run. They're mitigating it in future by putting more aggressive safeguards in their tooling, which is the correct way to mitigate this sort of issue.

Reply on news.ycombinator.com