Partial Cloudflare outage on October 25, 2022
blog.cloudflare.com
blog.cloudflare.com
Cloudflare CDN Partial Outage - https://news.ycombinator.com/item?id=33333332 - Oct 2022 (54 comments)
It was not DNS.
(You can obviously catch it with a blue-green deployment, which they do mention, but at that point some small portion of traffic has already been affected.)
Just a small suggestion: there's nothing in the post (unless I missed something) that mentions or even implies that this is not the "final report". You could have a small paragraph at the end saying "we might update this post further once we have more details", or even better, have a bullet point list of the things your team is still looking at. That way, readers of the blog post would know it might be worth it to come back later and read the post again.
They probably don't need to at all, but I can think or several ways it might unintentionally happen.
Perhaps someone chose to implement the modification as “separate headers & body, modify headers, put the request back together” and something went wrong that made the putting-back-together fail quietly (so the now malformed request passes to the next part without an exception being raised) or not get called at all in some circumstances. Alternately a badly coded routine to remove a header, perhaps the instrumentation information after the point in the processes it is no longer useful, maybe removed more than it should.
How such things get past a code review without someone suggesting a less error-prone method might be a question that gets asked in the internal investigation. I assume somewhere like CF has many checks between initial code modifications and things dropping into production, unless they've really taken “fail fast” to heart!
A change should be rolled out to a small fraction of nodes and then monitored for an extended period. 5% of requests returning errors would easily be spotable. Only when a new release has run stable on that portion for some time should it proceed to a bigger subset. You probably want to do this in several steps at the size of CF.
We also learn that it took customer reports to get the investigations rolling but 5xx errors are easily monitored so it points at internal monitoring being lacking even though it's hard for me to believe that they don't have an eye on this already.
It's not the first time that a deploy has brought CloudFlare (partially) down. From the timeline we see there's several hours between incident investigations starting and the rollout being stopped. The rollback should be the first thing considered even before looking at what the actual issue is.
Ideally you have someone sitting next to a red rollback button during a rollout whose only job is it to have an eye on all automatic and customer error reports. :)
2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.
2022-10-25 17:28: Peak impact with approximately 5% of all HTTP requests resulting in an error with status code 530.
2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers.
2022-10-25 18:04: Rollback is complete in all Upper Tiers.
2022-10-25 18:30: Rollback is complete.
The 503 graph appears to show a steep drop between 17:38 and and 17:45, which looks like the "rollback" speed to me.
I think they likely drained many of the lower tier caches/pops/edges after which roll back speed doesn't matter very much since the machines being rolled back likely weren't servicing traffic.
> large data centers acting as Upper tier for many customers.
To me this sounds like a very awkwardly worded way to say "we shifted traffic".
Clarification on whether traffic was shifted or tiers were drained would be nice.
2022-10-25 14:39: Multiple teams become involved in the investigation as more customers start reporting increases in errors.
2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.
How does cloudflare track their code releases and does cloudflare annotate their 4 major graphs (errors, rps, utilization, latency) with lines showing when things were pushed to prod?
> At first glance, the errors looked like they were caused by a different system that had started a release some time before.
And with somewhat noisy error rates I'm not sure even annotating graphs with releases would have made it unambiguous.
That does not surprise me. There are companies working with CI/CD systems whose tests easily take more than an hour. And until they are completed, they cannot start releasing (enforced via system permissions). This is a good thing, but the possibility of an emergency rollback should be considered beforehand and maybe (it always depends) exceptions made for such purposes.
And sometimes it makes more sense to revert slowly to make sure it doesn't get worse.
> Once identified we expedited a rollback which completed in 87 minutes.
I appreciate that it's a massive infrastructure spanning the entire globe, but... 87 minutes to revert a change? I wouldn't want the job of fixing it, but that doesn't seem good enough when the impact is that bad.
So if any caches were cleared in the process of rolling back, a slow rollback makes perfect sense to me.
I would expect a company like cloudflare would have previous revisions of software on their machines that could be reverted to near instantly.
Another thing to consider is what capacity they run at and how long a service takes to initialize. If you can only take 10% of your machines out at any given moment and each one takes a while to initialize, you don't really have much choice in the matter.
Lastly, if you look at the 530 graph, it seems that the drop in 530's was almost instant.
> 2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers.
This leads me to believe they shifted traffic away from the low tiers, which enabled them to roll back as slowly as they like at the cost of a likely minor increase in latency.
If their processes require a suite of tests be run on staging systems before the revert hits production, that may take considerable time depending on how much is potentially affected.
Also the 87 minutes might be for everything, perhaps it was a rolling update: update some nodes, test, update the next batch, repeat. This might mean that a good chunk of the infrastructure was updated for sooner and traffic could be diverted through those parts (presumably those nodes are over-specified, or just easily scaled, to be able to take a glut of extra load - as they would need to in more normal circumstances in order to contend with waxing & waning traffic patterns).
Perhaps it is even the case that the errant update didn't get out to all areas before the problem became apparent, so they were not rolling back over the whole network. Again this may offer the opportunity to divert load to non-affected areas while the others are rolled back.
So while the full revert took 87 minutes, things may have been up and running properly (except perhaps a little slower than usual) for most users/processes much quicker than that.
1) I wonder if they so many layers of CI/CD and testing, that the whole pipeline, even rollbacks probably re-run their tests which takes time. Are CI/CD runners the problem or it is something like network/DNS thing?
2) What if someone inside Cloudflare glibly yells "Why can't we just rsync our code to all data centers in less than a minute?" What would be the pushback this person would get? Why is it a stupid suggestion? Answering that would reveal some crazy layers of abstractions that get piled up in large companies.
Of course, anything else is probably gonna have worse UX, because it won't have all the infra pre-built and auto-configured for you.
Also you'll need at least one publicly-accessible server to be the Internet-facing side of it, and that part would necessarily not be free, unless you can get by on some free-tier VM somewhere.
I live on the same block as the Cloudfare headquarters. They have a big common space that has open windows to the street. They have some big projectors that play what are basically Screensavers on the wall in this common space, visible from the street. For months now one of these big wall sized projections has had in large white letters "this copy of office is not legit, please contact Microsoft to purchase a legit license of windows " (I paraphrase a bit). This seems like something they should address, no?
I bet cracked versions of Office don't display that message, so it's an indication that this one is actually legit :)
Either of these courses of action seem like even more of a public shame, which was not my intent.
Edited to add - I did, the night guard didn't seem to care or even know who to bring the issue to.
Roll forwards and back should usually be done slowly and carefully.
However, during a major outage, there is a good reason to do the rollback very fast. Ie. Within 1 minute. Fast rollbacks end any outage sooner. And if your service is already down, you have nothing to lose and everything to gain.
That provides an interesting infrastructure problem. Fast rollbacks probably involve restarting a large number of servers, having a large number of cold caches, etc. You can find new bugs that only rear their heads during a fast rollback. Things like docker registries getting overloaded, kubernetes control plane being overloaded, other services becoming unhealthy due to a large influx of requests, etc.
The traditional approach is 'dont take risks, don't do a fast rollback'. But that can double or more the length of your downtime. Especially if you roll back the wrong thing first.
I'd therefore encourage preparing ahead of time for fast rollbacks. The dev/test environment should automatically do 'big bang' rollbacks every week. Production rollbacks should even be practiced from time to time, getting faster and faster each time, keeping an eye on metrics to look for instability.
They had a 5% request failure rate at the peak of this outage. So a botched rollback could definitely do more harm than good if the impact isn’t 100% understood.
And faster monitoring, they detected the issue after 4.5 hours and took one to fix