Some S3 eng accidentally dropped a big chunk of the servers that were the s3 equivalent of an hdfs nameserver, ie mapping blob name to location info, as part of an unrelated config change.
While attempting to recover, the s3 team discovered and/or decided the nameserver needed a full restart. That's when they discovered the info in the nameserver had grown so large since the last full restart years previous that it took far longer than expected to restart the nameserver. Right around that point in time my guess is they realized just how shit their morning was going to be. And their afternoon.
Somewhere in there, they realized that their health dashboard depended on s3 working.
Though to be fair, as an aws customer, we -- along with the rest of internet -- were well aware that stuff was badly broken.
I feel terribly for whoever did this, because IIRC, he or she just fat fingered part of a command in a standard playbook, and the config script had no safeguards. I personally took down a company you've heard of in the exact same way; I knocked all pops off the internet because the config script had a hard requirement around certain values that was neither communicated to me nor checked. And I was trying to figure out wtf I did to a system that I was not particularly familiar with while receiving forwarded texts from the CEO about cascading datacenter down alerts.