Surprised it both took them so long to decide to rollback and that the rollback lasted so long.
That does not surprise me. There are companies working with CI/CD systems whose tests easily take more than an hour. And until they are completed, they cannot start releasing (enforced via system permissions). This is a good thing, but the possibility of an emergency rollback should be considered beforehand and maybe (it always depends) exceptions made for such purposes.
And sometimes it makes more sense to revert slowly to make sure it doesn't get worse.
> Once identified we expedited a rollback which completed in 87 minutes.
I appreciate that it's a massive infrastructure spanning the entire globe, but... 87 minutes to revert a change? I wouldn't want the job of fixing it, but that doesn't seem good enough when the impact is that bad.
1) I wonder if they so many layers of CI/CD and testing, that the whole pipeline, even rollbacks probably re-run their tests which takes time. Are CI/CD runners the problem or it is something like network/DNS thing?
2) What if someone inside Cloudflare glibly yells "Why can't we just rsync our code to all data centers in less than a minute?" What would be the pushback this person would get? Why is it a stupid suggestion? Answering that would reveal some crazy layers of abstractions that get piled up in large companies.
So if any caches were cleared in the process of rolling back, a slow rollback makes perfect sense to me.
I would expect a company like cloudflare would have previous revisions of software on their machines that could be reverted to near instantly.
Another thing to consider is what capacity they run at and how long a service takes to initialize. If you can only take 10% of your machines out at any given moment and each one takes a while to initialize, you don't really have much choice in the matter.
Lastly, if you look at the 530 graph, it seems that the drop in 530's was almost instant.
> 2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers.
This leads me to believe they shifted traffic away from the low tiers, which enabled them to roll back as slowly as they like at the cost of a likely minor increase in latency.
If their processes require a suite of tests be run on staging systems before the revert hits production, that may take considerable time depending on how much is potentially affected.
Also the 87 minutes might be for everything, perhaps it was a rolling update: update some nodes, test, update the next batch, repeat. This might mean that a good chunk of the infrastructure was updated for sooner and traffic could be diverted through those parts (presumably those nodes are over-specified, or just easily scaled, to be able to take a glut of extra load - as they would need to in more normal circumstances in order to contend with waxing & waning traffic patterns).
Perhaps it is even the case that the errant update didn't get out to all areas before the problem became apparent, so they were not rolling back over the whole network. Again this may offer the opportunity to divert load to non-affected areas while the others are rolled back.
So while the full revert took 87 minutes, things may have been up and running properly (except perhaps a little slower than usual) for most users/processes much quicker than that.