Let's Encrypt API v2 Service Disruption (12 Sep 2021)
letsencrypt.status.io
letsencrypt.status.io
September 12, 2021 20:30 UTC
Service disruption
September 12, 2021 05:39 UTC
Production planned maintenance
September 12, 2021 02:33 UTC
Degraded API and OCSP Performance
September 20, 2021 16:30 - 17:00 UTC
API Database Maintenance
Ah, that's unfortunate. It's hard to get RDBMS reliability/performance "right" in the face of changes, and scaling it isn't easy. But at least all the problems are well-known.
Aah yes... that magical "zero-downtime" cloud, lots of promises are made by the cloud providers in that area but all eventually broken at some point. Its a fact of life with technology unfortunately. :)
No need for the cloud with ScyllaDB[1], CockroachDB[2] and others all being used in production.
LetsEncrypt chose their setup for perfectly valid reasons they described on their blog[3]. Please describe your expertise for saying they're "doing it wrong", because its clear from their blog they put a lot of time thinking about it.
[1]https://www.scylladb.com/ [2]https://www.cockroachlabs.com/ [3]https://letsencrypt.org/2021/01/21/next-gen-database-servers...
But for what it's worth, just because they thought for a long time doesn't mean they made the right choices. Some of their design decisions have been not great. You can still create perfectly valid domain certs for domains you don't control with a variety of common attacks. Then there was the ACME API vulns. Either they knew about all this and chose a simpler design that's more flawed on purpose, or all that thinking just didn't end up being correct.
They might not be perfect, but you get what you pay for, just like open-source software. Free but no warranty. Fine by me since Let's Encrypt is available 99.999% of the time, outages are the exception rather than the rule (plus you should always be renewing your certs in good time and not at the last minute).
https://www.scylladb.com/2021/03/23/kiwi-com-nonstop-operati...
If the Cloud provider messes up, the customer still has to think about it. I'm not a huge fan of high level cloud services for critical infrastructure, because they're likely to go wrong (as high level services do from time to time) and when they do, you'll usually have little visibility into them, which makes recovery dependent on the Cloud provider.
I'd rather have potentially less uptime but more control, but maybe that's me.
For me the major consideration is getting away from toil so I can spend time improving things that will move the needle. If somebody else can do a half-decent job, and there's a backup plan in case things go wrong, please somebody else do it for me!