One thing I’d be interested to know more about is why it took 17 minutes to fix. While you can and always should strive to make them less likely, outages are inevitable, so how you respond is crucial. Here the outage was very obviously caused by a deployment that I’d assume was supervised by humans – why did it take 17 minutes to roll back?
Especially when you consider that they are getting DoS attacks every 2-3 minutes - so all deploys are going out into a hectic world and the dots maybe aren't that easy to connect under those circumstances.
- shit is not working
- is this an attack?
- no it's us
- how?
- that's how
- let's go back
- have to get supervisor
- roll back huge thing
really that long?
- monitoring system picks up irregularity (smoothed over some window of time, which delays alerting)
- alert propagates to humans
- humans may take time to notice alert (even a page takes a few seconds to read)
- humans make decisions, may need to talk to other humans (all of what you said above)
- humans evaluate correct procedure, double-check it (you don't want them making the wrong "fix" and making something else worse, do you?)
- humans execute commands
- commands take time to run on large collections of computers (running them completely in parallel can cause thundering herd issues, in some cases)
Things change in a time when you would freak getting up in the morning and say google.com did not work.
Automated processes can only mitigate so many edge cases. Even then, humans need to be involved, and that slows things down.
Another example - I can record a video and broadcast internationally, translated on the fly into dozens of languages, effectively communicating with significantly more people than if I could not harness that technical capability. In 2000 that communication process would have been orders of magnitude longer. (Did they have on-the-wire translation then? Idk, just making an assumption to illustrate my point.)