Wow. That is a truly impressive way to handle a security bug and I know it’s not the first time Let’s Encrypt has responded extremely quickly.
I would love to hear how their engineering practices make this possible.
The workforce is spread across the states. When you're drawn to the mission like I am, late nights here and there don't matter at all. I communicated with my wife and I'm sure others informed their significant others what was going on etc and why Friday, Saturday, Sunday, Monday, and Tuesday would be thrown out of wack. Members of the team put in much more hours than I did and that is truly impressive. It takes all of us with our different specialties to make an accurate and effective response.
Some of the things we do are internal post mortems and find ways to prevent the issue from happening again by either improving alerting/monitoring, writing a runbook, fixing code, and fixing misconceptions about a part of the entire system. We do weekly readings of various RFCs, the Baseline Requirements, and other CAs CP and CPS documents to again better understand our system and Web PKI as a whole. This is an understatement, but we heavily rely on automation. From the moment the call was made to stop issuance, an SRE was ready to run the code that disables the issuance pipeline.
The biggest takeaway is that communication and leadership makes all the difference.
I have to go, there's work to be done.
That is, I'm OK with being woken at 0200 to try to understand and if appropriate fix or recover from a disaster only so long as if I'd suspected this might happen the people expecting me to be awake at 0200 would have given me the resource (money, people, whatever) to fix it. If I feel like I don't have that support, I'll only start looking at your disaster during my working day.
My impression is that ISRG pays a lot of attention to preventing disasters, so if I worked for ISRG (not very practical since they're based on the US West Coast and I live in England) I'd be comfortable taking a call in the middle of the night to fix things.
Basically I keep a phone and laptop on me at all times.
This is in comparison to a friend that works somewhere that always has daily on call incidents that are not actually problems 95% of the time. That would piss me off even if I weren't always on call.
At huge scale there just will be incidents every day, if you're SRE for Google then every day is a bunch of problems - but at that scale those are routine incidents and you can afford the follow-the-sun team (actually I think Google SRE found it was better not to have true follow-the-sun but just two groups) to handle them, leaving only still rare "San Francisco just fell into the Bay" level incidents to actually wake people up.
We have an on-call rotation and a system for getting others notified and online quickly when necessary. We make sure not to bring too many people online so that some people are fresh and can rotate in later if the incident lasts longer.
It's not often that staff have to put in time at night or on weekends, and when it happens we work hard to make sure the problem doesn't happen again.