I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!
But once it happens things will change and, to be honest, likely for the worse.
edit> fix typo
Succession?
The sheer level of technical competence of your engineering team continues to astound me. (Yes, they made a mistake and didn't catch an error in the diff. But your response process went exactly as it should, and your postmortem is excellent.) I couldn't even begin to think about designing or implementing something of this complexity, much less being able to explain it to a layperson after a failure. It is really impressive, and I hope you will continue to do so into the future!
Most of the companies I've worked for unfortunately don't use your services, but I've always been a staunch advocate and converted a few. Maybe the higher-ups only see downtime and name recognition (i.e. you're not Amazon), but for what it's worth, us devs down the ladder definitely notice your transparency and communications, and it means the world. I've learned to structure my own postmortems after yours, and it's really aided in internal communications.
Thank you again. I can't wait for the day I get to work in a fully-Cloudflare stack :)
Not as transparent as "post it on the internet" but at least better than the usual hand wavey bullshit
Additionally, these post-mittens work for Cloudflare because they have a great reputation and good uptime. If this were happening daily or weekly, it would be a warning sign to customers.
It’s a strategy other companies could adopt, but to do it effectively requires changes all across the organization.
1. On the buy side, crappy big companies with procurement etc. may not have “actual engineers” deciding things. Maybe for something like Cloudflare that’s likely to sit within an actual technical team’s mandate
2. Not 100% foolproof but if a startup is selling its tool and has an uptime page and details of all its downtime and as a prospective customer you go and see that they have 3-4 hour downtimes once or twice a month, it should raise alarm bells.
IMO it is a slippery slope to see this as opportunity too strongly. Sure, doing the right thing may be net beneficial to the business in the long run...but the $RIGHT_THING should be done first and foremost because it's the right thing.
I think their calculation (to the extent you can call it that) is that in the interest of PR and damage control, it’s better to get a thorough postmortem out quickly to stem the bleeding and keep people like us from going “I can’t wait to hear what happened at Cloudflare” for a week. Now we know, the customers have an explanation, and this bad news cycle has a higher chance of ending quickly.
Incidents - yes. But why would a post-mortem turn someone off? The incident happened regardless. Do you think anyone would be more likely turned off by reading how they solved it / plan to prevent it on the future than by silence?
Don't companies have a fiduciary duty to calculate things; the reason for doing something actually cannot just be that it's a nice thing to do? Not down to the word, but at least the general decision to be this way?
Doing incident response (both outage and security), the tactical fixes for a specific problem are usually pretty easy. We can fix a bug, or change this specific plan to avoid the problem. The search for conditions that allowed the incident to occur can be alot more time consuming, and most organizations I've worked for are happy to make a couple tactical changes and move on.
There is not. From here there's an ongoing process with a formal post-mortem, all sorts of tickets tracking work to prevent further reoccurrence. This post is just the beginning internally.
> This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically.
They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that takes a day to put up the blog post, but doesn’t unnecessarily extend downtime because they are sprinting.
At around the same time, I was watching HBO’s The Wire. One scene had a high-profile shooting with a frantic police response; people were running everywhere trying to help. The lead commander on the scene gave this instruction to his sergeant: “Slow this thing down to a crawl. Give these bastards no chance to fuck up in a meaningful way.”
And then it hit me. That’s exactly how they ran the calls. I asked them about this, and they said absolutely – human nature, when things are broken badly, is biased towards action. You want to try to make a change, to reboot a system, to do something to hopefully make things better. But often if you were not careful, you risk losing information about the outage you are in. Best case scenario you luck into fixing the problem and don’t know how. Worst case? You’ve changed the state of an already broken system, and have done nothing but add more variables to unwind.
So now, every time I’m on an outage call, I try to do what Wes and Tim and Major Rawls would all do: I take control, pump the brakes, and make sure that we are capturing enough information about the current state that we don’t confuse ourselves further.
And with nowhere near the detail level of what was presented here. Typically lots of sweeping generalizations that don't tell you much about what happened, or give you any confidence they really know what happened or have the right fix in place.