It's a good thing we have ever-shrinking certificate lifetimes and automation never breaks. That's what I've been told, anyway.
It will forever be known as The Great Oops.
There are a few things that can cause tremendously widespread outages, essentially all of them network configuration changes. Actually deleting customer data is dramatically more difficult to the point of impossible - there are so many different services in so many different locations with so many layers of access control. There is no "one command" that can do such a thing - at the scale of a worldwide network of data centers there is no "rm -rf /".
The possibility Google will either manage to unleash a malicious AI on their infrastructure and/or develop a way to destroy a lot of data at scale quite efficiently or some combination of the two is far from zero.
Bear in mind, this "Little Oops" should also have been impossible: https://www.techspot.com/news/103207-google-reveals-how-blan...
"We deployed this private cloud with a missing parameter and it wasn't caught" is as different from "we wiped out all customer data" as hello world is from Kubernetes.
No one promised this "should be impossible". Did you confuse "we'll take steps to ensure this never happens again"?
You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner.
The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes to all of them.
You only need a command to rm rf one box, if you are distributing that command to every box.
Now sure, there are tons of security precautions and checks and such to prevent this! But pretending it's impossible is delusional. People do stupid stuff, at scale, every day.
The most likely scenario is a zero day in an environment necessitating an extremely rapid global rollout, combined with a plain, simple error.
But it can happen, and it only has to happen once. (Also FYI, telling me your work history just tells me you've drunk the koolaid, ain't proof you know more.)
The remediation was painful but it was not data loss.
It's the sort of thing that used to keep me up at night.
Break your control plane, and you can't stop the propagation of poison.
Propagate the wrong trust bundle... everywhere.
Also, it's not about the delete command. It's about the automatic cleanup following behind it that shreds everything, or repurposes the storage.
If didn't back it up yourself, it is gone forever.
Though I'm sure the major players are all over this risk which is why it hasn't happened.
1. Sales volume was lowest on weekends so if something went wrong it would affect fewer customers.
2. If something went wrong and I needed to revert, nobody was at work on weekends so it would not disrupt coworkers.
3. I always made it so reverting would be easy.
4. Most of my weekends were just relaxing at home, mostly doing online stuff (games, reading, videos) or doing offline stuff at my computer (programming my personal projects). It wasn't much of a bother at all to have an ssh open to something at work monitoring the new deployment for problems for the rest of Friday night and Saturday.
Frankly even with no CA redundancy, downtime would have to drag on for weeks to actually disrupt renewals. ACME certs usually get rotated after about 2/3rds of their duration has expired, so the upcoming 45 day certs will still have about 15 days of wiggle room.
https://zerossl.com/documentation/acme/
> By using ZeroSSL's ACME feature, you will be able to generate an unlimited amount of 90-day SSL certificates at no charge, also supporting multi-domain certificates and wildcards.
(Or an unrelated failure, of course)