Cloudflare outage on July 17, 2020
blog.cloudflare.com
blog.cloudflare.com
# Just be ready
Most definitely not the first time Cloudflare has had trouble, just like any other system: it will fail eventually. If you're complaining about the outage, ask yourself the question: why were not you prepared for this eventuality?
Spread your name servers, and use short-TTL weighted CNAMEs, defaulting to say, 99% Cloudflare, 1% your internal load balancer. The minute Cloudflare seems problematic, make it 0% 100% to bypass Cloudflare’s infrastructure completely. This should be tested periodically to ensure that your backends are able to scale & take the load without shedding due to the lack of CDN.
# Management practices
Cloudflare's core business is networking. It actually embarrasses me to see that Cloudflare YOLO'd a BGP change in a Juniper terminal without peer reviews and/or without a proper administration dashboard, exposing safe(guarded) operations, a simulation engine and co.? In particular, re-routing traffic / bypassing POPs must be a frequent task at scale, how can that not be automated so to avoid human mistakes?
If you look at the power rails of serious data centers out there, you will quickly notice that those systems, although built 3x for the purpose of still being redundant during maintenance periods, are heavily safeguarded and automated. While technicians often have to replace power elements, the maintenance access is highly restricted with unsafe functions tiered behind physical restrictions. An example of a common function that's safeguarded is the automatic denial of an input command that would shift electrical load onto lines beyond their designed capacity - which could happen by mistake if the technician made a bad assumption (e.g. load sharing line is up while it's down) or if the assumption became violated since last check (e.g. load sharing line was up when checked, became down at a later time - milliseconds before the input even).
The output is from a Juniper router, not Cisco.
We'll get there, but it's no surprise CF isn't doing this today; it would put them waaaay ahead of the pack if they did.
Would not catch weird firmware quirks in the real hardware, definitely would've caught this fat-finger typo.
Have you done this before? I'd be interested to hear how those conversations went.
If I had to guess, I'd say it's because network engineers simply don't need / get this know-how on normal scale. Most SW developers on the other hand are not very good (good enough for CF scale) at networking. Which leads to networking guys doing their thing the way it was always done... (hope I didn't offend anyone, just guessing)
I hope they strengthen their dev department... I know I'd love a challenge like that. :)
Or some kind of tool that processes the resultant routing tables to generate some kind of "route usage" for every given link and device, maybe even feed it with a table of expected traffic to given destinations.
Except if you're using CF for DNS service, this wouldn't have worked, as both CF's website & DNS servers were impacted by the outage.
This got me Googling, and best as I can tell, CF doesn't support zone transfers. (They support being a client, but not a server. So, they could function as one's secondary system, but not as the primary.)
Then I'm stuck with a bunch of bad questions about how often to poll, and whether CF's rate limits would support any reasonably quick poll interval.
(The big problem is that we have other tooling that relies on being able to update DNS, the big one being ACME for certificate renewal. The changes it makes to CF would need to be rapidly replicated out to the nameserver.)
(Nothing in the post really strikes me as particular to CF, either. I think I could easily replace everything I've said here with "Route 53" and end up in the same bucket, maybe plus or minus zone xfers working.)
In all seriousness, I wonder if they are going to have to change the marketing on the site now...
However it is a bit bad of CF that a single configuration error can bring all the slave servers down. It mean that they have no redundancy in term of BGP misstakes. Customers of CF that want to avoid this would benefit to add an additional slave server outside the hand of CF.
Zonemaster (dns sanity checking tool) actually complain about CF hosted domain names because of the lack of AS redundancy. The outage yesterday demonstrate nicely why that is an concern and why one should care. https://zonemaster.iis.se/?resultid=7d1fab165987e195
We use AWS + Azure + GCP (yes, all 3) as our authoritative NS and keep them all in sync with octodns.
If your service does scale at first place, then you don't need Cloudflare most of the time.
Sure, anyone can scale my localized infrastructure for the traffic of 100,000 IP Cameras. Can anyone do it for 10,000,000 pwnd devices? Sure, but it'll likely start not being so practical without multiple POPs. Do I want to hire a dozen network & threat detection engineers to build/maintain that, complicate my processes, and pay for the infrastructure moving forward for a once-in-a-year event? Not really, no.
The way I see it, Cloudflare acts just like an insurance policy. Pay for a fraction of the actual cost, get your back covered, and profit from the expertise when it hits the fan.
DDoS mitigation providers wanted absolutely absurd amounts. Cloudflare took me on for $200 a month (I had confirmed beforehand). Mitigated all the attacks. All tickets were responded within minutes by network engineers working to mitigate the attack.
How do you justify the cost? I'm seriously asking - I have had a hard time making this pitch myself, I am curious if you have (recent!) experience with this.
No ? Then the costs are not justified, and while it would be better from a tech perspective it makes no business sense.
Yes ? Well then you spend X to save Y, with Y being greater than X, so it's an easy sell as long you don't start with "cloudflare is never down" (which is not true).
It’s worked surprisingly well.
Are you thinking of a cloud-computing context here? Seems to me a lot hinges on this, but perhaps I'm misunderstanding you.
If so, this would answer the scale question, and would presumably translate into increased prices until the incident is over. (I'm assuming CloudFlare offer a cheaper solution than doing it yourself on a cloud.)
If not, and you own the physical capacity yourself, wouldn't you do away with CloudFlare entirely?
Cost could be an issue. We had something similar (not in the same context) in a company I worked for before. We could shift traffic, but that would cost 2-3x more, so it was not the preferred path unless we had problems.
It surprises me that many (big) companies did not learn the lesson already. We had a similar thing happening already years ago with dyn in 2016 (https://en.wikipedia.org/wiki/2016_Dyn_cyberattack), and it was surprising how many companies relied on a single DNS provider.
Presumably this is a function of scale? At a certain point it's going to be worth running your own CDN.
Which can't be done because it invalids the point of using CloudFlare!
CloudFlare is used to protect your site from DDoS attacks and ransoms. It has to hide the IPs of the servers otherwise attackers will DDoS the servers directly, bypassing CloudFlare.
I know of at least one site that works this way which has allowed them to weather CDN outages.
>
> Cloudflare's core business is networking. It actually embarrasses me to see that Cloudflare YOLO'd a BGP change in a Juniper terminal without peer reviews and/or without a proper administration dashboard, exposing safe(guarded) operations, a simulation engine and co.? In particular, re-routing traffic / bypassing POPs must be a frequent task at scale, how can that not be automated so to avoid human mistakes?
We don't know if this was entirely the case, based on the timeline for the initial incident that prompted the change gone awry, there very well could have been an ITIL-Style CR created and processed within this time.
Judging by the edits made, this wasn't just simply taking a POP out of service entirely, but reducing the amount of (or eliminating all of the) traffic from neighboring POPs sent to compute at the ATL location. I can't image that this exact type of change is all that common. BGP anycast actually makes things significantly more complicated when removing edges.
As far as the mechanics go, with junos's CLI, there's not a lot of difference between what the intended command would have been, and the one that actually happened.
---
What they probably wanted
| example@MX1> configure
|
| {master}[edit]
| example@MX1# edit policy-options policy-statement 6-BBONE- OUT
|
| {master}[edit policy-options policy-statement 6-BBONE-OUT]
| example@MX1# deactivate term 6-SITE-LOCAL
|
| {master}[edit policy-options policy-statement 6-BBONE-OUT]
| example@MX1# commit
---
What might have happened
| example@MX1> configure
|
| {master}[edit]
| example@MX1# edit policy-options policy-statement 6-BBONE- OUT
|
| {master}[edit policy-options policy-statement 6-BBONE-OUT]
| example@MX1# deactivate term 6-SITE-LOCAL from prefix-list 6-SITE-LOCAL
|
| {master}[edit policy-options policy-statement 6-BBONE-OUT]
| example@MX1# commit
---
Initially, this seems like quite a bit of difference, however, Junos has a hyperactive autocomplete that triggers on spaces,. that deactivate could have been as short as "dea ter 6 fr p 6"
I'm not aware of any routing simulation product that is able to simulate complex bgp interactions, and report on effective routes of simulated traffic, as well as CPU load predictions. The closest I am aware of is running GNS3 (or a bunch of VM routers) overnight and capturing SNMP.
On the other hand, automating these kinds of changes would seem trivial, but such a service would have to be as fault tolerant as any other project, but is most certainly a worthwhile endeavor especially since integration is actually relatively easy, Junos provides some nice REST and XML APIs on the management interface that can do pretty much everything the CLI can, except start a shell.
It is difficult to write automation to cover all the tasks you would do, even if you cover the ones most commonly done, you will have higher risk on the rest.
A linter or higher level instruction set which well tested may be better solution perhaps. Automation if any perhaps should be after that ?
Nailed it.
It is also that typically even in large organizations companies with the money and people, fire drills and DR drills go the same way, it is known there is going to be drill and people react accordingly. Chaos Monkey style testing/drills rarely happen.
I would say building resiliency to your architecture is the key to this. Just like having a single customer > 50% revenue is enormous risk for any business , relying on any single service provider is also enormous risk . In manufacturing it is common to insist on second source for a part, IBM did that to Intel for the PC which is why AMD got into x86.
In this case a proper HA would serve better - minimum of 2 CDN networks always sharing 50% of the load and the have capacity to double if required. If they cannot scale that much then distribute to 3-4 and keep traffic no more than 25-35% per provider , such that a loss one means only additional 10%-20% traffic to the rest.
Also it is important that two service providers should be actually different, if they both depend on the single and the same ISP or backbone to service an area, it is not going to be effective.
The principle should apply across the entire infra Name Servers, CDNs, load balancers, Storage, compute, DBs, Payment g/w and registrars ( use multiple domains example.com example.io each with one registrar).
>> just like any other system: it will fail eventually
An analogy combining these two points: one could pay a million dollars AN HOUR to the top software engineers alive on the entire planet… and at some point you will encounter failure. Technology and humans are both fallible, end of story. This is why SLAs exist with specific uptime targets to meet, and reimbursements should that SLA be broken. Anyone who believes the rare outage is unacceptable: fine; bring that layer in-house and pay engineers millions a year to do the best they can. You'll still encounter failures–and likely more of them.
The usual cry you'll hear from some business is "we lost $x million during the downtime!". Yes, and without some company like Cloudflare in front of your business, you'd probably be losing $x million multiplied by orders of magnitude you don't even want to imagine.
"You can't have your cake and eat it too."
I only wish they could update cloudflarestatus.com more quickly. Shouldn't there be some mechanism to update that immediately when there is an incident? When the entire internet knows your down and your status page says All Systems GO! it looks very poorly on them.
> Because of the way we securely connect to StatusPage.io from most locations where our team is based. The traffic got blackholed in ATL, keeping us from updating it.
(it's statuspage.io's status page)
Guess I wasn't wrong, they might even have lost access to internal monitoring systems which is pretty unfortunate in such a situation. If you ask them about Cloudflare Access they will happily tell you that it was built for internal tool access and that they use it for everything, later they went on to sell it as a product.
Now I'll admit that proper change management won't catch every issue, but the issue described in this post seems like something that should have been caught. It's a little worrisome that a company that so much of the internet relies on is apparently playing fast and loose with major config changes. The changes you describe in the post-mortem sound like they will fix the immediate problem and possibly prevent future occurrences of this exact same problem, but what about making broader changes about how you deploy these things?
Not only that, but this single config change not only brought down the CDN, but brought down both 1.1.1.1 and the secondary 1.0.0.1. Was this type of failure never tested, or...? What's the point in having both if they both go down at the same time?
In an ideal world, yes there would be higher level tools to do everything and checks and balances for everything, but I assume half the tools are like our standard Unix cli tools - all too powerful and the operator better know what they are doing
By the time they have all the right tools for everything, the company would have become a dinosaur disrupted by some other technology or company
Even in an "emergency" situation like described in the post-mortem (which I'm actually confused about, because Prince's tweet said it was actually "routine maintenance", which makes it even less justifiable), there should be a standard playbook with pre-tested and pre-approved failover options to enact. There definitely should not be someone just manually fingering in a config file on the fly.
>In an ideal world, yes there would be higher level tools to do everything and checks and balances for everything, but I assume half the tools are like our standard Unix cli tools - all too powerful and the operator better know what they are doing
This is fine for some startup still hacking their way through an MVP, but for a global, multi-billion dollar company that prides themselves on supporting a significant part of the global internet, "the operator better know what they are doing" is unacceptable and not good enough. There should be multiple layers of protection against this stuff, so that even if an operator is having a nutty, they don't break half the internet.
>By the time they have all the right tools for everything, the company would have become a dinosaur disrupted by some other technology or company
I can't agree at all. We're talking about some basic change management and testing stuff here. I've been at plenty of organizations (even ones much smaller and with less resources than CF) that have gotten this right, and they certainly are not dinosaurs nor have they been "disrupted". Testing and change management is as fundamental as encryption or password hashing. If you don't have it, it's not because it's hard, it's just because you haven't bothered to implement it.
Let's be clear: I'm not talking about some small subset of fantasy companies. It is a strict compliance requirement for many of the world's largest industries that you must have things like change management processes, incident response playbooks, and testing in place. A "myth" it certainly is not. Again, it's a fundamental thing no different than the most basic security requirements that we also expect (which is no coincidence, as one of the components of many security frameworks is proper change management controls). I'm not saying it has to be bulletproof and perfect (in fact, it never will be), but you still have to have something.
Put another, possible more easily understood way: "don't test in production" is a requirement (some times even a legal requirement) for most companies once they reach a certain size. It's not just a funny saying that devs joke about.
That can't will into existence engineers who are actually thoughtful, creative, and skilled at those things. They can't make the test suite actually anticipate the bugs that will be written. They can't make the runbook have correct steps with no unwanted side effects for every failure mode. They can't make the reviewer see what the author doesn't.
Two, maybe three people in my 100-person org are legitimately good at this stuff. No amount of grandstanding about what a serious business we are changes that. In fact the people most enthusiastic about compliance and process usually end up undermining the real, substantive technical quality that the compliance process was aiming at.
However, the point remains that things like change management aren't some nebulous, pie-in-the-sky concept. That they are in compliance frameworks speaks to the fact that they are very fundamental and basic (compliance frameworks are typically the bare minimum of things you should be doing).
A half-assed CM process won't catch all errors by any means. But it will catch the most basic ones. And according to CF's post-mortem, the initial issue that kicked off the chain of events was "backbone congestion", which has got to be one of the most basic things that would be included in an IR playbook (Prince even referred to it as "routine maintenance" in an earlier message). And then the config change that was put in place also seems like a fairly basic change that probably would have been caught with with some basic testing.
That's what catches my eye the most. This incident wasn't some wild, niche, couldn't-have-been-predicted event. It was "backbone congestion", followed by an attempted change of routes on the backbone to alleviate the congestion. For an internet services company to not have a standard, pre-approved and pre-tested solution for resolving something as predictable as backbone congestion is shocking to me.
I would say we should ready between the lines here. Most postmortems are marketing artefacts to some sense designed to reassure customers so taking them at face value is not really good idea, I have written and read enough to know that it is rarely ever the full story for variety of reasons.
It is quite possible like you say they are playing fast and loose with config and they have systemic process risks as you point out.
It is also possible that the issue is not just routine congestion i.e. it is symptom and not the root cause, and they are not talking about it in detail either to make the communication simple enough for non networking experts to understand, but has enough depth for technical folks to kind of follow along and reassure their managements that cloudflare has handle on it . It could also be revealing more may give their competitors some crucial information about their IP/architecture and they are calling it routine congestion to not reveal that.
Lastly, all of the above are not mutually exclusive, it could be all of them in parts too.
The teams I've seen be the most comfortable with these knowably bad processes have been the strongest advocates against guardrails. The cost to your reputation and bottom line when something does go wrong is opaque to many engineers, but these kinds of ridiculous errors do lose customers.
Edit: A common saying where I work is that there are no stupid mistakes only bad processes. The first step to rooting them out is admitting that they're process level issues.
These are not the people I want dictating my workflow.
That doesn't mean it's the "enemy of success." If we could raise its status to attract people with genuine technical competence and empathy for the realities of building, we could do a lot better.
>Edit: A common saying where I work is that there are no stupid mistakes only bad processes. The first step to rooting them out is admitting that they're process level issues.
Human stupidity is unbounded; such a process asymptotically approaches "do not change production ever." If you're a monopoly with a money printer, then keeping it running probably is more important than any improvement you could make. If you're under competitive pressure, you need to ship, and shipping necessarily involves risk.
One possibility would be to parse the config file and do a rudimentary simulation, taking into account traffic volumes, and warn the user if the result would be expected to overload any routes.
Another possibility would be to do something a bit smarter than just instantly, blindly propagating a change to every peer at the same time. If the bad routes had been incrementally rolled out to other routers, there might have been time for someone to notice that the metrics looked abnormal before Atlanta became overloaded. (I don't know whether that's feasible with BGP, or if it would require a smarter protocol, but it seems like it would be worth looking into.)
Finally, it seems like if a config change is made to a router and it immediately crashes, there should be systems that help to correlate those two events, so that it doesn't take half an hour to identify and revert the change.
They wrote a two-page essay and buried in the middle is the one-sentence root cause of "someone changed something and made a mistake". No explanation or details given.
This is exactly how to pretend you have incident reports without actually having them.
Luckily, the HN crowd is able to piece it together and explain in detail.
- No performance benchmarks for rules despite using a backtracking regexp engine.
- No canary deployments, allowing the mistake to propagate to their entire network at once.
A mature engineering organization is very unlikely to make such basic mistakes, because it has in-depth peer review for both design and implementation.
Of course, their openness about the incidents is commendable and is the only reason why my company hasn't dropped them a long time ago, but it doesn't excuse such an easily avoidable downtime.
Immediate hot flash.
After I got off the call (thank god he had an appointment to run to), I checked it out. Our internal dashboards were all green so we realized it was a DNS issue pretty quickly.
Since we couldn't get into Cloudflare we searched Twitter and realized it was their issue and I stopped worrying.
One of the benefits of CF and other major internet vendors is that when they're down, you can kind of shrug it off and tell your customers to wait a bit. Not so if you're using a smaller/unknown CDN company.
It's not DNS
There's no way it's DNS
It was DNS
And if everyone else on the AWS / CF is down as well, then it is no one's fault. We all keep calm and just wait it out.
It's the same as the old phrase about no one ever getting fired for buying IBM.
At the end, your customer, assuming it is a SaaS, just need an excuse for "their" customers or the End User. And there is nothing better than a big household name's fault so no one get the blame.
So the whole thing is written off, everybody is in the clear, and everyone can get on with their other business. :)
I often feel sad for people that have not developed the critical thinking skills needed to look past marketing to where the realm of logic and reason resides
The simple reality is that if your site is down because AWS is down then it means a lot of other sites are down too. Which means you don’t look anywhere near as bad as you would if you were the only site not working.
Logic and reason is all well and good but human perception is a very real thing that businesses need to keep in mind. It isn’t always entirely logical.
At most one region was impacted, and it is easy to be multi-regional in AWS, in fact that is kinda of the point
Further this comment is in service of the moronic axiom of "No one ever got fired for buying <<insert large company>>" my response to that has always been and will always be "sure they have and they should"
It is simply not true that buying AWS, IBM, Cisco, or any other large vendor is complete insulator from all responsibility to maintain reliable systems nor should it be
Any administrator or developer that is making buying choices based on that is not an person I would like to ever work or do business with
To extrapolate out to the original point: if multiple top ten traffic web sites are having issues (and this has happened once or twice in the last few years due to AWS or other cloud issues) then your site being down is less notable in customers minds.
The grandparent was talking about a SaaS service so then I would expect the customers to be technically minded people.
If you are selling to masses then sure, but if you are selling me a Line of Business SaaS service then no I do not care if facebook and reddit was down, that is not relevant to how the SaaS product should be running
For instance Basecamp could be down due to AWS. And Basecamp ( and in this case Hey as well ), but are SaaS. And their customer may not always be technically minded. Given the usage of these tools, while inside tech circle, are not only used by technical people.
And At the end of the day it is all about trade offs.
I also dont think anyone ever make a purchase decision purely on brand or not fired for X. For example, despite AMD offering lower price and offer more core and performance. Server Vendors hasn't all switched to AMD at once. In fact, Intel Server still has months of backlog order to fill. This isn't simply because Intel is better connected with Vendors, it is the fact most of those End user / customers are still demanding Intel CPU. Because it is well tested, with more specific libraries, tools, guarantees and support. Many of these factors cant be quantitatively measured, and therefore would only be judged when the final price difference are shown. In this case No one gets fire for using X is another phase for if it aren't broke, dont fix it.
Install new host -> Migrate VM;s live to new host -> Shutdown old host
Zero Downtime
This is simply not possible with a Xeon to Epyc Migration which requires downtime, as well as testing of the guest to ensure nothing weird happens
it is a prime example of Vendor Lockin.
IF there was away to live migrate with zero downtime Epyc would own the datacenter today
I’ve had colo equipment that ran with 5-nine uptime for years eventually get unlucky and be down for an hour and it was “all my fault”. Switch the service to Amazon which achieves much worse uptime, but now it’s “well if Amazon is down, what can you do?”
Frankly when something seen as core internet infrastructure goes down, the measly SaaS companies pretty reasonably don’t take any blame.
To be clear, often these are services where there isn’t the engineering budget, nor honestly the need, for multi-cloud and geographically distributed multi-master services.
But the plan B in this case is switching nameservers, which we could certainly have done (and briefly considered), but it could be error prone and would take longer for those changes to propagate than it would for CF to fix the issue, most likely.
The best option was simply inaction, if that makes sense. There are times when not doing something is a better idea than doing something.
Tangential: nobody got fired for buying IBM
Taking accountability and having backup plans are extremely important, but you simply can't remove every last shred of dependence. You eventually have to accept that there are things that are out of your control and may take you by surprise despite best efforts.
Other than that it's your choice whether to make your infrastructure dependent on a bunch of unreliable centralized SPOFs from big corporations or build highly available infrastructure relying on servers from many different providers running your own DNS servers with DNS routing, failover, etc. You will definitely beat Cloudflare's availability this way many times over.
What if a political event impacts you, for instance? A pandemic? A storm taking out a major data center? A weird Linux kernel edge case that only happens beyond a certain point in time? That only sounds ridiculous because it hasn't happened, but weird things like that happen all the time. There are so many unseen possibilities.
I understand that might sound unreasonable or facetious or like I'm expanding the scope.
The point is, the more confident that you've built something that has no SPOF the more exposed your are to the risk of it, because one probably does exist.
I remember when I first deployed DNS routed system it was too reactive, constantly jumping between servers, monitoring was too sensitive, it didn't wait for servers to stabilize to return them into the mix and exponential backoff was taking servers out for far too long. But even given all that it was still able to avoid outages caused by data center failures and connectivity problems.
> If you design for resilience, you get more resilience and you build confidence as you see the evidence how the system works in real world.
You simply can't foresee or eliminate all risk. This is referred to as "the turkey problem." It's not my idea, but one I certainly subscribe to.
https://www.convexresearch.com.br/en/insights/the-turkey-pro...
Speaking of things that don't make sense... if it's unforeseeable, one will have a difficult time adequately preparing for it
Famous to the point of being a cliche, the titanic was thought to be unsinkable, and I would have a similarly hard time convincing the engineers behind the ship's design to believe otherwise.
The level of confidence you're displaying in predicting the unforeseeable is something you may want to take a deeper look at.
I understand that most of those leetcode corporations don't care much about resilience, likely even incapable of producing highly reliable systems, and may give you a false impression that reliability is something of an unachievable fantasy. But it's not, it's something we have enough research done on and can do really well today if needed, we are not in titanic era anymore.
I have high confidence in these things (not in "predicting the unforeseeable"), because I've done them myself. My edge infrastructure had like half an hour of downtime total in many years, almost a decade already.
show me a time where all of AWS was down in every region all at the same time
> Last week we made a server-side code change that triggered crashes for some iOS apps using the Facebook SDK. We quickly resolved the issue without requiring action from developers.
Certainly, cloudflare, could be lying to us. First, this seems super unlikely to me, but in the event that they are, it’s still a situation which would literally cause the described issue if it did happen. Therefore it can still be learned from in the same way. Again, it feels unlikely for it to be a lie considering the specificity and lack of other viable explanations.
Look, most of mistakes are silly or combination of silly. We think it is good to understand them, but in reality someone on the team probably pointed out that this can happen and was ignored, as it wasn’t a priority. And the biggest motivator to make sure companies prioritize uptime is to tell them, that we don’t care about their excuses, we care about uptime.
I wonder whether github still on MySql waiting for another outage.
Read the reason - human error. As old as humans, all I learned is that Cloudflare doesn’t have sufficient automations and checks in place.
We can reasonably assume that the postmortem is truthful. Since they’re a publicly traded company, lying about this incident would be a quick way to turn an embarrassment into a felony.
Some seasoned sysadmin will say to me, "Of course it's always from a configuration change. What else could it be?" I don't know, it seems like there are other possible causes. But in today's superautomated infrastructures, maybe config files are the last soft spot.
I'm not even sure.. Is that second sentence supposed to signal some sort of success? Dropping 50% of your traffic isn't isolated. If your gonna try to spin it, at least bury the damn lede. Further:
> The affected locations were San Jose, Dallas, Seattle, Los Angeles, Chicago, Washington, DC, Richmond, Newark, Atlanta, London, Amsterdam, Frankfurt, Paris, Stockholm, Moscow, St. Petersburg, São Paulo, Curitiba, and Porto Alegre. Other locations continued to operate normally.
Locations with THEIR equipment, but certainly not all "affected" locations. I live 4 hours from Dallas and can assure you that I was impacted. That coverage is like.. Most of the United States, Europe, Brazil and who knows how much of South America? Oh right, 50% of their traffic!
Just in the past year Verizon, IBM, Apple, now Cloudflare have seen outages from BGP misconfiguration. The Verizon issue took down a significant part of the internet.
BGP is a liability to society. We need something which doesn't constantly cause widespread outages.
Any other tool that allows you to announce connectivity will have the same problem. In this scenario it was still even legitimate routes, it just was too much for their specific link to handle.
It is entirely possible to cause a similar problem with OSPF or (lol) IS-IS, with the “right” misconfiguration and route metrics.
During the most recent outage a few weeks ago, Azure were available to discuss the issue by phone. I wish I could say the same for Cloudflare.
I would be interested to hear from anybody who knows of a good alternative to Cloudflare. I'm completely fed up with them.
Obviously, AWS and GCP offer their own CDN systems as well. (CloudFront and Cloud CDN, respectively)
There are tons of third party CDNs as well.
Unless by "good alternative" you mean that you're on Cloudflare's free plan, and hoping to find someone else who will willingly soak up huge amounts of bandwidth for free?
Cloudflare is one of the only services I know of that offers this, but it's hard to complain about a short outage every once in awhile when you're paying nothing or very little. The Cloudflare customers who are paying quite a bit are surely upset.
No we are not using Cloudflare's free plan.
I was simply interested to know if anybody else had recommendations for a Cloudflare alternative.
Introduce a maximum-prefix limit on our backbone BGP sessions - this would have shut down the backbone in Atlanta, but our network is built to function properly without a backbone. This change will be deployed on Monday, July 20.
Change the BGP local-preference for local server routes. This change will prevent a single location from attracting other locations’ traffic in a similar manner. This change has been deployed following the incident."
It should be noted that configuring prefix limits for your BGP peers is kind of BGP 101. It's mentioned in every "BGP Best Practices" type document.[1] It's there for exactly this purpose to prevent router meltdown and resource exhaustion. For a company who blows their horn as much as these folks seem to about their network this is embarrassing.
I think it's worth mentioning that it was this time last year when Verizon bungled their own BGP configuration and brought down parts of the internet. When that incident occurred Cloudflare's CEO was front and center excoriating them for accepting routes without basic filtering [2]. This is exact same class of misconfiguration that befell them yesterday.
[1] https://team-cymru.com/community-services/templates/secure-b...
[2] https://twitter.com/eastdakota/status/1143182575680143361?la...
foo.example.org. CNAME blah.customer.cloudflare.whatever. with a ttl of loke 5 minutes
Then when cloudflare goes down, you switch that to your origin server, or your static system is down page or something. Most of your traffic moves within 5 minutes and when you're satisfied Cloudflare is working again, you move traffic back.
If you've delegated your domain to Cloudflare, you can switch your delegation at your registrar, and a lot of TLDs update their servers pretty quick, but the TTL is usually at least a day, so you'll be waiting a while for traffic to move.
Towards the end of the game, a CDN the bbc used crashed, taking a million people’s live streams offline.
Traditional TV with its 20 million plus viewers worked fine.
A 15 minute global outage during the World Cup or Super Bowl is not acceptable in the world of boring old TV
Meanwhile github has been down how many times this year?
IT is still a terrible industry of cowboys. It’s just hidden under the veneer of abstaction, microservices and outsourcing. Other industries like the national grid or water of radio have outages that affect a local geographic area or a limited number of people, but they are far more distributed than the modern internet. It’s ironic a network designed to survive nuclear war can’t survive a typo.
https://m.huffingtonpost.co.uk/entry/bbc-iplayer-crashes-in-...
From the timeline in the blog post the issue with Atlanta was fixed between 21:39 to 21:47 but a status page wasn't published until 21:37. Everything had been broken for over 20 minutes at that stage with lots of people already posting about it or other status pages reflecting issues. See https://twitter.com/OhNoItsFusl/status/1284239769548005376 or https://twitter.com/npmstatus/status/1284235702540984321
Without an accurate status page it leaves businesses pointing the finger everywhere wondering whether its their hosting provider having issues, their CDN, DNS provider etc etc.
I'm currently working on a project to monitor all the 3rd party stack you use for your applications. Hit me up if you want, access I'll give free access for a year+ to some folks to get feedbacks.
They slung some mud not long ago and now it came to bite them. They were a bit righteous on their reliability. However, anyone in this game long enough knows it’s only a matter of time before shit goes down. If they didn’t have any graybeards over there to tell them this, then hopefully they earned some gray.
Stuff was down long enough for Googles and OpenDNS caches to expire, and to take down DigitalOcean in some respects.
Thankfully CF can afford to learn and make improvements for the future. Not all organizations are that lucky.
Oh, at whom?
Half of the internet
Or is BGP intrinsically a unsafe protocol without builtin protections on this sort of human mistakes?
It's a dynamic, constantly-changing system, and the effects of your actions may not always be seen - it's not always obvious how other networks behave and will react to your route announcements. And so even trying to snapshot the current state and say "hypothetically, would this change be a bad idea?" can be hard to get right.
Now, this particular case was of internal BGP use at Cloudflare, so everything I said doesn't necessarily apply... but it still does a little. Even internal networks can be so complicated, they may as well be un-analyzable.
I think the problems here are pretty deep, unfortunately, and have to do with our "network of networks" design.
Of course, this particular outage would not have been prevented even if they had used
# commit confirmed
Even as a lowly network engineer working on networks much, much smaller than Cloudflare's, for pretty much my entire professional career, it's been my standard practice to start off pretty much ANY change -- no matter how trivial -- with
# commit confirmed 3
# reload in 3
This applies even when making a changes that are so "simple" that they just "can't" go wrong or have any unexpected or unintended effects.
In fact, it applies ESPECIALLY in those case! It's when you let your guard down that you'll get hit.
---
Fortunately, all that was necessary (I assume) to recover in this case was to
# rollback
I would hope that Cloudflare has full out-of-band access to all of their gear and are able to easily recover from mistakes like this. If they had lost access to the Atlanta router and weren't able to log in and revert the configuration manually, this outage could have lasted much, much longer.
BUT
What's interesting here is that so many non-CloudFlare services went down (including even AWS - partially) caused by DNS outage - because every sysadmin and his mom are using 1.1.1.1 as their DNS.
Decentralization is what makes the internet function in a robust fashion.
https://investors.fastly.com/news/news-details/2020/Fastly-A...:
> By emulating networks at scale, Tesuto’s technology can be used to create sandbox environments that simulate the entire Fastly network, providing a view into the potential impact of a deployment cycle before it is put into production.
What is really scary though that half of the internet stopped working. That's not ok!
I'm surprised stuff like this wouldn't happen more often and there would at least be a well tested, automated remediation step in place which also validates the change prior to going live.
I get they may be busy solving other issues, but it's interesting this isn't a more fool proof procedure given the huge impact a mistake can have.
I am not suggesting to ditch DoH and go back to unencrypted DNS.
There are a number of small independent and trusted DNS providers who support DoT. Uncensored DNS is one that I woul absolutely recommend.
The Cloudflare revolvers definitely went down (1.1.1.1 and 1.0.0.1), do we know if authoratative DNS did?