Grave flaws in BGP Error handling
blog.benjojo.co.uk
blog.benjojo.co.uk
A few years ago a team of researchers tried to use a feature attribute flag in BGP. Their experiment caused an important BGP software to crash.
The BGP community reacted by thanking the researchers for uncovering that flaw and worked on improving stability. No, just kidding. They shouted at the researchers and asked them to stop, which they eventually did.
I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers.
Sources: https://mailman.nanog.org/pipermail/nanog/2018-December/0984... https://mailman.nanog.org/pipermail/nanog/2019-January/09876... https://mailman.nanog.org/pipermail/nanog/2019-January/09918... https://mailman.nanog.org/pipermail/nanog/2019-January/09914...
I suspect a Network Operators mailing list is going to be full of people trying to keep production up, with the tools they've got, rather than the tools they'd like. Thinking the software quality on the routers they're using are dogshit is probably just called "Monday" and thinking their peers' routers are even worse is called "Tuesday".
I can understand they don't appreciate "research" done on production systems that might cause issues when they already know their equipment sucks.
Bitching at researchers (note: not "researchers") who are finding and calling out those flaws before someone really bad comes along and exploits them is somewhere between unhelpful and negligent. Internet security is always improved by more sunlight.
A researcher could have bought their own copy of the lock and picked it there. Or even borrowed yours with some notice, or let you know so you could have them use an interior door instead, or any number of other things that didn't break your actual house.
We call out vendors when their quality suffers but you only have two or three practical vendor options for high end routers depending on your use case/feature requirements. All three have varying cycles of hardware and software quality, with hardware and software rarely being in phase for the same vendor.
The fact that all three vendors make most of their money on non-network focused enterprises with less sophisticated engineers doesn't help get development attention where it needs to be for global infrastructure.
I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. When "fixing" a router means you break half the internet, you just don't fix it. This isn't some walled garden where you can just say "tough shit if you broke, fix it".
Most of the "issues" these researchers find are well known, but backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.
The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed. Once a vulnerability is known then someone out there is going to start exploiting it almost immediately.
> backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.
Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP. This is a security lesson that has been learnt in recent years by many sectors of the IT industry and, judging by the woeful response that Ben Cartwright-Cox describes in his post, it's one that the network backbone is going to be learning soon.
Research is great but if you're poking around breaking things via DoS in the internet you're going to be judged as a bad peer regardless what the route daemon does. The security of the internet constantly moves forward but that doesn't mean the expectation of improving has to hold more precedence than the expectation of running. It's still very much a human to human system for all but the actual route exchange.
So you've never worked in a corporate environment. Here's how that conversation would go:
Hey guys, some researchers found out if you use a test flag meant for the lab on the public internet, it breaks all our BGP sessions.
OK, so drop their feed and block them?
Great, done.
>Once a vulnerability is known then someone out there is going to start exploiting it almost immediately.
And yet the "vulnerability" in question was known, and was not being immediately exploited if you read through the mailing list or were participating in NANOG at that point in time. So your statement is provably false.
>Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP.
And yet we're having this conversation in 2023, they have operated the same way for 40+ years, and somehow the internet is still working. Bad actors get blackholed, it worked in the past, it'll continue working in the future. The reality is that backbone routing is expensive, and expecting everyone to update their kit on YOUR timeline isn't reasonable.
I am prophesying that a similar reckoning is likely to come upon the network backbone. You're arguing that the cost of entry to the game (BGP peering) is such that the old ways will continue to work. Let's hope you're right.
Practically, I accept that there is a real difference. ASNs are much harder to acquire than IP addresses, and there are far fewer of them. That difference might mean that blocking continues to be an effective mechanism. However, in a world where malicious actors are increasingly just nation states wearing a flimsy mask, I'm not sure that the difficulty of getting access to sufficient numbers of ASNs is going to be a forever mitigation.
The main thing I have taken away from this thread is that when it comes to vulnerability management, network operators seem to be pretty far behind the curve in mindset terms. The arguments being made in here ("you can't break other people's stuff", "we have a lot of boxes and SLAs mean we can't patch quickly") aren't even slightly new. The well-proven reality is that threat actors don't give one single shit about any of that.
> Great, done.
The "great" is the problem, no?
Bearing in mind we also couldn't just "apt-get upgrade" in place, most boxes required hard reboots to apply the patches.
The answer to your question is no, as per a few others have said, Don't violate Rule #1. We see bad actors very often, our job is to keep the bits flowing and the internet online.
Keeping the internet online is painful enough as it is without "researchers" dropping thousands of routers to "prove a point."
Does any sane person think that 1 week is enough time to both notify every user of FRR in the WORLD, as well as ensure they have upgraded their installations for a "bug" that only is triggered by someone using experimental flags meant for testing purposes only?
Mind you, these researchers didn't test ANYTHING in a lab environment first "because it's a lot of work to test all the open source BGP implementations in the wild". AKA: I'm lazy.
That's a gross exaggeration: they said that they first tried it "in a controlled environment with different versions of Cisco, BIRD, and Quagga routers" [0] without issue. Presumably, the problem we know about would have been avoided if they had happened to include FRR in their test suite beforehand. But that raises the next question, exactly how many implementations does one need to test before one is no longer "lazy"?
[0] https://mailman.nanog.org/pipermail/nanog/2019-January/09876...
1) Beleaguered people misbehave.
2) You are usually the architect of your own destruction.
3) It’s easier to see other people’s problems than your own.
Stir into a death spiral and let it rest for six to twelve months. Bake at 375° for two years, then serve cold with a sprinkling of schadenfreude to offset the bitterness.
Overall I agree with your message though, and of course things shouldn't stop working just because you use attributes reserved for development.
[0] https://mailman.nanog.org/pipermail/nanog/2019-January/09876...
Thou shalt not blow up another's network knowingly.
Blowing up another's network because you didn't test your theory in your own isolated lab space first counts as blowing up another's network knowingly. Because what the hell else did you think would happen?
Cripes.
- I think it's OK not to want experiments like these to be conducted connected to the global internet and rather in a lab such as the one benjojo used.
- NANOG is known to be a bit of a joke within the networking space, not too long ago someone complained about NTP issues with traffic passing through TATA and they were told to make their devices stratum 0 or something to that effect, so don't always take them too seriously.
I think the first applies here.
Not saying your summary of that thread is wrong, I just wanted to expand a bit to maybe clarify:
- someone posted to NANOG about issues with their (European) servers reaching 0.freebsd.pool.ntp.org
- they noticed they were being routed through Tata (AS6453)
- they mentioned that their other servers (e.g. in Africa) had no issues
- they posted to NANOG to ask about this
- they thought it was a routing or FreeBSD issue (they later posted about the issue to a FreeBSD list as well)
- the NTP pool uses GeoDNS to map clients to servers
- the OP's server IP address was incorrectly geolocated to Mauritius
- the Mauritius zone in the pool has only one server
- so the OP would only ever get that IP as an answer
Full thread here:
https://mailman.nanog.org/pipermail/nanog/2023-August/222706.html
From there, that sub-thread was mostly about NTP attacks and GPS security and mounting receivers on top of datacenter buildings and such.
Counting up the messages (if my mail client's threading capabilities are to be trusted), of the 94 or so total messages in the thread, 79 (84%) of the messages were in that GPS-related subthread, and only 15 messages responded to the OPs issue directly.
I didn't really see this as a failing of NANOG per-se, it's just a public mailing list and some folks went on a nerd sniping tangent (cf. XKCD #356).
Dropping the whole session, especially in a DFZ scenario is really bad. The churn that you generate is immense for all your neighbors.
GREASE means that when you build your "advanced" security device which "protects" customers by treating everything different as hostile, it won't work, so you'll need to tweak it to at least just ignore such differences as irrelevant, which is enough that we can come back later and intentionally improve things.
Previously, without GREASE, we'd have to guess what oversights we could exploit to avoid this "protection" to deliver protocol improvements, if we guessed wrong nothing works, or everybody's security is broken, sometimes both. e.g. for TLS 1.2 the oversight we found is, if you're resuming an existing session the "security" products just wave that through because otherwise they break people's real workflows. So in TLS 1.3 protocol as actually spoken essentially an initial connection goes like this:
Client: Hi some.dns.name.example I'm a TLS 1.2 Client, I'd like to resume our previous conversation #randomNonsense. Also, completely unrelated, I happen to speak FlyCasualThisIsReallyTLSv1.3 and so I have these TLSv1.3KeyAgreementParameters.
then either:
TLS 1.3 Server: Hi Client, of course, let's continue from there. [ whereupon everything further is encrypted because this is actually TLS 1.3, but to a dumb middlebox it makes sense that they're just resuming a prior encrypted conversation #randomNonsense which it doesn't remember ]
OR
TLS 1.2 Server: Er, no I don't remember any such conversation and I don't know FlyCasualThisIsReallyTLSv1.3 so let's start a fresh conversation as normal.
If you are, good on ya. Continue doing $deity's/Bob's work.
If your system is on the internet it is now a potential piece in a potential global war.
We've blasted past that age at this point, do everybody gets to don their robe and wizard hat, pick up the staff, and yeet patches, routing rules, and packets betwixt one another.
BGP is one of the most robust protocols out there: there's no state (as in protocol state), you can treat each message independently, you can extend it super easy and so on.
And because you can treat every message independently, you can have a really secure and stable mechanism to deal with any badness that might happen.
The outrage here is understandable. I would be pissed if I was paged because halfway across the world, some asshole caused this intentionally. Also this mailing list is aimed at the operators, not the vendors. Different communities.
The vendor response here, I do agree is irresponsible. Besides OpenBSD, every one here just sucked.
Or they felt like some of us feel about Tesla/Musk unleashing cars with "Full" "Self-Driving" on public roads.
We’ve been building trillions of dollars of public transit since about the same time as we started building the National highway system. So when exactly does public transit start saving all these lives?
A new train from SF to LA has been in the works since 1996. They are expecting partial completion for $30 billion by 2030 which is projected to carry ~15 million passengers * ~400 miles = 6 billion passenger miles per year.
The fatality rate for passenger cars is ~0.57 per 100 million miles, so eliminating 6 billion passenger miles could be reasonably expected to save about 60 * 0.57 = 34 lives a year out of the ~40,000 that are dying on the roads annually.
This is just rough napkin math showing the scale of the problem would require a roughly $30 trillion investment in public transit to appreciably shift passenger-miles away from passenger cars.
Public transit can be very useful. But it’s clearly not the solution to eliminating driving deaths.
You argue that it would be very expensive to fix the problems we've made for ourselves in order to get useful public transportation and that's certainly true. It's also very expensive to invent self-driving cars and part that cost has already been paid with lives lost and human suffering, but that hasn't stopped us from developing those systems. We could decide to invest instead in improving our cities and infrastructure. We just choose to let obscenely rich people kill innocent Americans while they play with their profit generating spying car tech instead.
There are other people building driverless cars, who have safer systems that they rolled out reliably.
Musk isn't saving people from drunk drivers, he's trying to catch up with the competition and chose to kill people to catch up.
Still, I guess it keeps support contracts active and network engineers in a perpetual hell of on call alerts
The robustness principle is also long considered to be not that great of an idea for protocols. There's a good document here going into this:
> With that being said, I would like to thank the OpenBSD security team, who very rapidly acknowledged my report, and prepared a patch. My only regret with dealing with the OpenBSD team was reporting the issue to them too quickly, as they are uninterested in waiting for any kind of coordination in disclosure.
Lesson to network engineers: switch your routers to OpenBSD/OpenBGPd and keep up on your syspatches to escape this particular hell. ;)
This BS with disclosure coordination and all the other crap vendors have been trying to train researchers to do for years needs to stop. You find a issue, you publish it, vendors need to deal with bad code and products in realtime, not when it suits them.
There is IMHO no ethical alternative to responsible disclosure, and using actual people who can get harmed by having their information disclosed as pawns isn't the way to go forward.
One of the main reasons for coordinated disclosure is that, over time, it has become evident that vendors will not do that. We can say it until we're blue in the face, but vendor behavior will remain largely unchanged regardless (there are a few bright spots, but not nearly enough).
So you go to war with the army you have. In this case, that means frequently deferring/coordinating security event communications. As much as I wish the market would appropriately penalize vendors that encourage researchers to sit on 0-days, that rarely happens, and the cost of insecurity is passed onto consumers instead.
Tl;dr delaying disclosure does suck, but you should still do it.
'Tis the world we live in, so might as well get used to it.
You could technically replace all the control and decision-making with whatever off-the-shelf computer platform you want, and have it drive the carrier-grade router and "manually" populate its routing table via some API/Telnet/SNMP. In fact, as long as the hardware packet-routing core is still good, this might be a good way to upgrade out-of-support equipment that is no longer getting updates to its control-plane computer.
It was only broken in the sense that it requires you to get out a keyboard and manually continue to a root prompt and type `fsck` (and then Y a bunch of times) and presumably some files may have been corrupted but i haven’t seen any.
For a router I’d much prefer that it comes up on its own after I lose power, but the system isn’t totally broken or anything. It’s happened probably 3 times in the last year or so.
To another commenter saying it’s crazy to not run a UPS with your router: this is residential. 99% of people don’t have a UPS for their router. They probably don’t even have a separate router and all and just use an all in one modem/router/wifi combo. None of these people are crazy, they’re just not made of money and a UPS is expensive. IMO it’s more crazy to not have a journaled file system in this day and age, no other systems have issues with sudden power loss at my house.
I wonder if the outage happened during patching or something maybe?
Say, compare RFC3174 (sha1) and e.g. rfc4664 (l2vpn) the first is almost a copy-paste implementation, with tests and all. the second reads like a "write a presentation, ten pages minimum"
or look at the base bgp4 rfc - that overspecified imaginary state machine is a good illustration of how bad things are there.
https://bgp.tools/as/266970?show-low-vis#prefixes
[bgp.tools Alert] for AS200676 A hijack on 2a06:a005:2730::/44 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured)
[bgp.tools Alert] for AS200676 A hijack on 2a0a:6040:b1f4::/48 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured)
[bgp.tools Alert] for AS200676 A hijack on 2a0a:6040:b1f1::/48 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured)
[bgp.tools Alert] for AS200676 A hijack on 2a0a:6040:b1f0::/48 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured)
BGP is a complete mess.
Also, for those responding that you don't see the problem, it's simple: remote unsanitized input cause denial of service
We made some thorough code review w.r.t. RFC 7606 compliance years ago.
“We found the majority of catastrophic failures could easily have been prevented by performing simple testing on error handling code – the last line of defense – even without an understanding of the software design.”
Effectively the impacted instances are now no longer able to do their job, they cannot propagate more rules.
That's the flaw.
When they receive something that they don't understand they need to propagate without breaking connection, and they do not.
And this is how things are done, per the article:
> On 2 June 2023, a small Brazilian network (re)announced one of their internet routes with a small bit of information called an attribute that was corrupted. The information on this route was for a feature that had not finished standardisation, but was set up in such a way that if an intermediate router did not understand it, then the intermediate router would pass it on unchanged.
> As many routers did not understand this attribute, this was no problem for them. They just took the information and propagated it along.
However, the information was corrupted at the source, and those devices that did understand the attributed detected the corruption and rejected it:
> However it turned out that Juniper routers running even slightly modern software did understand this attribute, and since the attribute was corrupted the software in its default configuration would respond by raising an error that would shut down the whole BGP session.
The question is whether the session shutdown was appropriate (at least by default).
I think Juniper's default is reasonable, as attributes can communicate important information about the intentions of the network operators, and rejecting a particular attribute and basically ignoring the intentions of the operators may also cause problems.
I also think that thinking that perhaps allowing the session to continue but throwing a (loud) warning is also reasonable (and there is a flag for that).
No they did not, they shutdown a whole session down for a malformed announcement. The right decision would have been to ignore the corrupt attribute and maybe not install that particular route.
A logged message saying "Attribute 129 with value 0 is invalid" would have helped operators more than just blindly shutting down sessions through which this was announced.
It is nothing like that.
HTTP requests do not propagate across an organization's entire network, or possibly the entire Internet.
> Or, for an even better analogy, it is like an HTTP server that, upon receiving a malformed HTTP request, drops all current and future traffic from that IP.
So just like (e.g.) fail2ban blocking IPs that continuously send bad login requests? That's awesome! I would want my web server to block bad clients.
Heck, it would be nice if web browsers would do the same thing with content they downloaded. In the early days of the web there were all sorts of garbage files out there and instead of forcing people to correct their HTML they tried to be clever in mind reading:
* https://en.wikipedia.org/wiki/Tag_soup
For a time there was an entire 'movement' to get people to clean up their act:
* https://en.wikipedia.org/wiki/W3C_Markup_Validation_Service
* https://en.wikipedia.org/wiki/HTML_Tidy
It's fine to have a flag to have stringency be a policy that can be toggled, but I think Postel's law ("be liberal in what you accept") can cause issues over the long-term:
Well, somewhat, except that it's not awesome. It's like fail2ban, but implemented on a server that sits behind a load balancer. So, when it receives a bad login request from an IP and it bans that IP, it actually banned the IP of the load balancer, so now it won't receive any other requests. So that a malicious client can just send 1 bad request, and take down everyone else using the whole server.
Basically, this whole discussion is not so much about Postel's law, it's about limiting failure domains. A bad route advertisement should make that 1 route inaccessible. It is not good design for a bad route advertisement to take down all the other good routes. It is particularly bad design for a protocol where routes are automatically propagated by devices that don't think there's anything wrong with them.
If you want to compare it to the HTML situation, the bug can also be seen as a browser that, when it receives invalid HTML from www.google.com/search would not only throw an error, but also refuse any other connection to *.google.com/*. The proposed fix is not to interpret anything as valid HTML. It is to show an error only when accessing www.google.com/search.
I think Juniper's default is reasonable: rejecting a particular attribute and basically ignoring the intentions of the operators could also cause problems, and instead of trying to read the minds of those running the networks, the software doesn't try to be clever.
I also think that thinking that perhaps allowing the session to continue but throwing a (loud) warning is also reasonable (and there is a flag for that).
My point still stands: attributes can have important information about the intentions of how a network operator wishes their network to be seen to the rest of the world. Having a corrupt attribute, and thinking it's okay to simply assume that without that attribute the rest of the advertisement is still valid is not fine IMHO: you're basically trying to guess/assume the intentions of the source of the advertisement.
For all we know ignoring that attribute and accepting the rest of the advertisement as-is could have caused flood of traffic over the connection causing a just-as-effective-DoS.
In the scenario that started off the author’s investigation, the bad attribute kicked COLT off the Internet even though COLT did not connect directly to the Brazilian ISP that sent the bad route update. When they received the bad update, COLT’s routers dropped their BGP sessions with intermediate ISPs that were unrelated to the Brazilian ISP, causing complete loss of connectivity.
If COLT’s routers had treated the error as a route withdrawal, they would have lost connectivity to the Brazilian ISP but not the rest of the Internet.
But! All your other routes that you advertised earlier with perfectly fine attributes will continue working. This is what prevents this from becoming a DoS vulnerability.
Let's take a more complete example.
ISP1 advertises 2 routes to ISP2:
35.67.0.0/16 attr1=val1
37.61.0.0/16 attr1=corr
35.67.0.0/16 attr1=val1
37.61.0.0/16 attr1=corr
78.0.0.0/8
After the fix, ISP3 will keep routing traffic to 35.67.0.0/16 (respecting the value val1 for attribute attr1) and to 78.0.0.0/8 through ISP2. It will not route traffic to 37.61.0.0/16.
Maybe switch off on request is a dubious design choice for infrastructure that you'd rather stay alive, but it sounds like a new version of the spec has already been released to make that an option.
If you shutdown the whole session because of one bad forwarded message, you are indeed a broken implementation.
If it were just a Juniper implementation flaw it’d be less interesting than explaining an aspect of BGP error handling more generally which is indeed what the article was, for those of us aware of BGP but not familiar with specific protocol design decisions and implications.
If you quibbled with the word “Grave” I might agree since it’s not industrywide and impacting major vendors like Cisco and Huawei and those implementing RFC 7606 (although there is no patch for Nokia or Extreme or Bird!) I’d counterpropose that a more neutral TLDR title/summary for an HN audience might be “Grave flaws in non compliant BGP error handling.”
But on balance I think the author is better off with raising the alarm with a stronger headline; he clearly put his passion to good use and I learned a little something as a result.
Absolutely not. To me, it's a miracle that the internet even works.