Hacker News Reader

Comment by dogleash | Hacker News Reader

> I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers.

I suspect a Network Operators mailing list is going to be full of people trying to keep production up, with the tools they've got, rather than the tools they'd like. Thinking the software quality on the routers they're using are dogshit is probably just called "Monday" and thinking their peers' routers are even worse is called "Tuesday".

I can understand they don't appreciate "research" done on production systems that might cause issues when they already know their equipment sucks.

View on HN

I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks.

Bitching at researchers (note: not "researchers") who are finding and calling out those flaws before someone really bad comes along and exploits them is somewhere between unhelpful and negligent. Internet security is always improved by more sunlight.

I suppose you have never seen, ever, in your professional career, the "C" suits making business deals and thus decisions on what's to be acquired, with big network/server/storage/database/software vendors, over the head and against the advice of those then called to support crap in production.

>I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks.

I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. When "fixing" a router means you break half the internet, you just don't fix it. This isn't some walled garden where you can just say "tough shit if you broke, fix it".

Most of the "issues" these researchers find are well known, but backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.

> I think you fail to realize that the internet is a bunch of disparate networks that no one person controls.

The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed. Once a vulnerability is known then someone out there is going to start exploiting it almost immediately.

> backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.

Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP. This is a security lesson that has been learnt in recent years by many sectors of the IT industry and, judging by the woeful response that Ben Cartwright-Cox describes in his post, it's one that the network backbone is going to be learning soon.

>The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed.

So you've never worked in a corporate environment. Here's how that conversation would go:

Hey guys, some researchers found out if you use a test flag meant for the lab on the public internet, it breaks all our BGP sessions.

OK, so drop their feed and block them?

Great, done.

>Once a vulnerability is known then someone out there is going to start exploiting it almost immediately.

And yet the "vulnerability" in question was known, and was not being immediately exploited if you read through the mailing list or were participating in NANOG at that point in time. So your statement is provably false.

>Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP.

And yet we're having this conversation in 2023, they have operated the same way for 40+ years, and somehow the internet is still working. Bad actors get blackholed, it worked in the past, it'll continue working in the future. The reality is that backbone routing is expensive, and expecting everyone to update their kit on YOUR timeline isn't reasonable.

I am aware that network operators have behaved this way for very many years. The point I am making is that all of the IT industry used to attempt to "work around" security vulnerabilities in the same way, until log4shell and all the others gradually beat that propensity out of them.

I am prophesying that a similar reckoning is likely to come upon the network backbone. You're arguing that the cost of entry to the game (BGP peering) is such that the old ways will continue to work. Let's hope you're right.

Software ecosystems like libraries and frameworks have completely different propagation and remediation mechanics compared to federated systems like core Internet backbone routers and switches is the thing. Try as we might conceptualize otherwise the modern Internet from a packet’s purview is more like a loose confederation of ultimately privatized or state-run fiefdoms than a cellular automata digraph explosion. So actors that try to act maliciously against the network will be basically shut out given the rule of an iron fist being the default.

Conceptually, there is no special difference between a remotely exploitable vulnerability in an HTTP server and a remotely exploitable vulnerability in a BGP router. It would have been theoretically possible to have dealt with log4shell by blocking every IP address that was sending malicious payloads, for instance.

Practically, I accept that there is a real difference. ASNs are much harder to acquire than IP addresses, and there are far fewer of them. That difference might mean that blocking continues to be an effective mechanism. However, in a world where malicious actors are increasingly just nation states wearing a flimsy mask, I'm not sure that the difficulty of getting access to sufficient numbers of ASNs is going to be a forever mitigation.

The main thing I have taken away from this thread is that when it comes to vulnerability management, network operators seem to be pretty far behind the curve in mindset terms. The arguments being made in here ("you can't break other people's stuff", "we have a lot of boxes and SLAs mean we can't patch quickly") aren't even slightly new. The well-proven reality is that threat actors don't give one single shit about any of that.

Operators could have raised this with vendors to fix.

> OK, so drop their feed and block them?

> Great, done.

The "great" is the problem, no?

Backbone operator who was effected by this. We had a large number of routers in production with this bug, we were aware and upgrading as fast as we could, but with 99.999% uptime SLA's we only have so many minutes per router we could afford for downtime/outages. We had schedules in place (approx 3 months of out-of-hours upgrades) 1 week warning was a bullshit move. Dropping the BGP sessions on 1000's of routers globally was stupidity.

Bearing in mind we also couldn't just "apt-get upgrade" in place, most boxes required hard reboots to apply the patches.

The answer to your question is no, as per a few others have said, Don't violate Rule #1. We see bad actors very often, our job is to keep the bits flowing and the internet online.

Keeping the internet online is painful enough as it is without "researchers" dropping thousands of routers to "prove a point."

Verbatim this is the same things people said in early 00's about people testing XSS et al. against poorly coded PHP/Perl sites.

The internet does not owe a negligent operator peering. No one person controls the internet but you sure as hell can find yourself kicked out if your goal is to be right on principals instead of a good peer. The only hubris which could be at play is someone thinking the rest of the internet should follow their desires instead of the other way around.

Research is great but if you're poking around breaking things via DoS in the internet you're going to be judged as a bad peer regardless what the route daemon does. The security of the internet constantly moves forward but that doesn't mean the expectation of improving has to hold more precedence than the expectation of running. It's still very much a human to human system for all but the actual route exchange.

> I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks

We call out vendors when their quality suffers but you only have two or three practical vendor options for high end routers depending on your use case/feature requirements. All three have varying cycles of hardware and software quality, with hardware and software rarely being in phase for the same vendor.

The fact that all three vendors make most of their money on non-network focused enterprises with less sophisticated engineers doesn't help get development attention where it needs to be for global infrastructure.

I practised picking locks on your house and broke the lock because of a poor lock design. Yeah okay the lock sucks. But I probably shouldn't have been doing it on your house in the first place and now you've got a broken lock. Who are you mad at?

A researcher could have bought their own copy of the lock and picked it there. Or even borrowed yours with some notice, or let you know so you could have them use an interior door instead, or any number of other things that didn't break your actual house.

"I can get into this lock" will not have the same effect or call to action as "I can get into your doors" but it admittedly has to be aimed at the landlords, not the doormen.

That is not really a matching comparison. With BGP, there is kind of public space (global BGP tables) and participants are free to attach valid attributes to their routes. From a cursory look, these researchers used announcement that was compliant with BGP standards (in contrast to Ben Cartwright-Cox from this article, who used fuzzed invalid attributes), so they did something that was generally acceptable behavior for BGP participants.

Rule number 1 of the internet is don't break other people's shit. You can do whatever you want on your own equipment but don't break other people.

Network operators are also the ones that pop into IPV6 discussions to voice their displeasure with having support another protocol.

To be fair… supporting dual stack really does suck

A lot about networking sucks, but that's what people are paying them for.

Also to be fair, IME there seems to be a lot more vocal pro-IPv6 people on the NANOG list than not.

Not really, ISPs (who are on nanog) tend to like ipv6. The ones who complain are sysadmins and network operators of campus/corporate networks.

They've only had over a decade to get their act together. I'm being generous, seeing IPv6 is older than the original IPv4 internet was when it went mainstream.

Except there's no "staging Internet" to test such equipment, and even if there was it would still sometimes not translate to the real open network. I'm sorry, were they inconvinienced with the world at large?

Reply on news.ycombinator.com