BGP and RPKI
aa.net.uk
aa.net.uk
As well as going above and beyond when it comes to sorting problems with BT, I appreciate that like with this post they are not afraid to stick their neck out and say what they think.
One thing it does show to me is that, even though the site was made available by Cloudflare only recently, my ISP has already acknowledged and responded to the post. Given that they have explained the situation, what it means, and that they are actively looking into it. The trust they built up with me since I started with them, means that I do trust that they are looking into this.
BGP as a whole does need a solution, and this may be one of a few ways to achieve this.
I am glad they are not rushing their solution so that my connection stays online while I have to stay isolated.
On the flip side, I do agree that some ISPs will drag their feet, and try to ignore everything for the sake of "money" or "can't be bothered" types of excuses. These are the ones that really should be pushed hard against to make the changes that are required. - I am just glad AAISP isn't one of them.
The choice to block one route to make isbgpsafeyet apparently pass is bogus, it's like silencing a fire alarm (which is also something I see far too many businesses choosing to do, oh Zone A keeps alarming, we'll just disable it even though now that means more danger, engineer calls are too expensive)
Those familiar with my work will know that I read a lot of accident investigation reports, and a recurring theme in maritime accidents involving large vessels is that the BNWAS (a technology which alerts sleeping crew elsewhere on a vessel that for some reason nobody on the bridge is paying attention, probably because they're asleep) is "somehow" muted, with its logging disabled or even entirely switched off during an incident. That might not seem like a big deal but big ships are slow and so these incidents often take place over many minutes or hours, thus there is plenty of time for somebody to wake up, realise there's a problem, wake someone else up to help fix it, and fix the problem before you smash the ship into an island clearly marked on your chart.
If Cloudflare has any sense they'll change the bogus route up every week or two so that outfits like A&A keep getting these "annoying" alarms and concerns from customers never cease.
> This is marginally better than routing to somewhere else (some attacker) but it still means a black hole in the Internet.
No. It's the difference between an inconvenience (routing problem which can be reported and then corrected by a human) and a grave security problem (data is being sent to completely the wrong place because you didn't bother with the mechanism to prevent that).
Also, this glosses over the _very common_ scenario where the bogus routes are more specific than legitimate ones. If AA doesn't have a route to one specific /24 containing secure.bankofamerica.com that has "somehow" been advertised by a Taiwanese ISP for the past hour, that data will be bundled along with the rest of the /16 or whatever else (BoA's assignment is old and a mess) to the Akamai boxes (I think) which actually serve it anyway. Rather than a "black hole" there was no problem except for the bogus advertisement, which RPKI would have helped block.
Both are therefore useful. RPKI protects from mistake and mitigate a bit the ability to spoof a route (adding the legit ASN makes the AS path longer and less preferred).
Weird angle. Unless the RPKI standard is somehow actively encouraging people to violate social distancing policies, I don't see any connection with Covid-19..
To me this whole article just reads like a network operator complaining that someone else is trying to hold them accountable.
In terms of our day to day lives it might feel like the proverbial month of Sundays right now, but for operations teams it’s more like an unending stream of Friday afternoons in terms of sensitivity to making big infrastructure changes.
We've known BGP's been vulnerable in this way for years, so it's a bit of a weird time to actively encourage people to publicly shame their ISPs for being "unsafe".
As you say, Cloudflare have been promoting RPKI for a couple of years now and it's disappointing that more of the big players haven't implemented it yet but is now the time?
1: https://blog.cloudflare.com/is-bgp-safe-yet-rpki-routing-sec...
> 17/04/2020, 4:00:00 pm BST
> Today, we are releasing isBGPSafeYet.com, a website to track deployments and filtering of invalid routes by the major networks.
On the other hand, AAISP started automatically assigning IPv6 addresses ~9 years ago, so you can hardly accuse them of dragging their feet. The OP was published on a Saturday, after all.
RPKI is no surprise. People have been beating on their upstreams for it for well over a year. Almost all Internet Exchanges have enabled BGP Origin Validation on their route servers (thanks to the efforts of folks like Job from NTT). It's about time we have a site like this that highlights the overall status of it. That said, there's more we can be doing here to provide metrics on RPKI adoption on the Internet.
My home ISP hasn't deployed IPv6 yet. Though, if they cited COVID-19 as a contributing factor when asked about it, I wouldn't be stunned...
This will allow them to pass Cloudflare's test page, without actually validating and rejecting invalid routes or even necessarily signing their own.
Saying things like "it's scaring our users", "others are not using it", "it's bad timing", "transit providers should be filtering", no actual non-emotional arguments why they aren't doing it and only shifting the responsibility to secure the internet. I'm too done with companies like that.
Not really though, they do agree in the post that something needs to be done, they just don't agree that RPKI is quite the right answer and that Cloudflare's fearmongering scaretactic is the right move to push for RPKI.
They also grab Coronovirus as a rationale for doing nothing right now:
"Since this has now happened a few times, we felt it worth giving some more information that may be useful to customers and others who've seen these tweets (either directed at us, or at other ISPs), explaining a bit about what BGP is and how RPKI can extend it, and also our feelings about Cloudflare attempting to build support in this manner, especially now, during the Corona Virus situation."
If you look at this NANOG thread [2] nobody is complaining about ATT announcing they have implemented RPKI. So is there a negative downside? No. Has CloudFlare pushed some carriers into an awkward position given they are showcasing the true state of carriers as it pertains to route security in BGP? Yes. Andrews & Arnold are trying to tell their customers that their safety is paramount. Yet, they don't have a timeline to address the problem that other carriers have spent considerable time implementing over the last couple years. So, while Andrews & Arnold may be a great ISP - are they above public disclosure of an area they need to improve? No.
I applaud CloudFlare for showing end users which carriers are not spending time and resources on doing their due diligence to protect their customers. Especially business customers who rely on their parent AS to operate their business safely. Andrews & Arnold's response is suspect at best given their subjective response to the "why" behind why they've chosen to do nothing.
Finally - beyond CloudFlare NIST has been publishing these statistics for much longer. Just because CloudFlare has shown light on the topic - does not mean they are the bad actor. There are plenty of other outlets that have been highly supportive of these deployments - NIST [3] and RIPE [4], among very vocal proponents.
So, after parsing the reality of the values of RPKI for a small amount of time - the question around why Andrews & Arnold have chosen to do nothing feels different and, in my opinion, even more appropriate. Beyond that their response feels very hollow and weak on the technicalities which have put them in a spotlight they'd rather not deal with right now.
[0] https://blog.cloudflare.com/rpki/ [1] https://blog.thousandeyes.com/visualizing-the-benefits-of-rp... [2] https://mailman.nanog.org/pipermail/nanog/2019-February/thre... [3] https://rpki-monitor.antd.nist.gov/#rpki_adopters [4] https://labs.ripe.net/Members/antony_stergiopoulos/results-o...
They don’t want to jump into rash decisions with minimal staff or staff dispersed across home locations and not able to work as effectively as normal - which could lead to broken BGP routes.
The easy response to your snark is "no, people are just going to use DoH".
Most of the egregious problems with IP options have been worked around well, again for decades.
ARP is problematic, but only within a segment. Your attacker has to be close. For IPv6, we have SEND, but it is not without problems.
> The easy response to your snark is "no, people are just going to use DoH".
DoH doesn't prevent someone from poisoning well-trafficked DoH resolvers to redirect individual domains. DNSSEC does. I run it on my zones, and it's not so bad... About 40% of resolver traffic to my domains is verifying, too, so we're getting there...
Yes, and DoH does nothing to protect queries between resolving servers and authoritative servers, unlike DNSSEC, which does protect those queries. Together, you end up with end to end protection.
Something like 30% of domains in .COM have DNSSEC, in part because of things like Cloudflare making it so dang easy. Yes, the top domains / services tend not to, and this is unfortunate. But we're well on our way.
And the possibility for poisoning worsens other attacks (it's still trivial in many cases to steal email with DNS cache poisoning... 25 years after I first played with the idea).
There's no reason not to deploy DNSSEC today. The DOH resolvers you're enamored with are almost all validating, so it's a way to get end to end validation right now.
And, of course, that 1-2% is almost entirely a bunch of random tiny zones nobody cares about. You can quickly discover that for yourself by feeding any list of the Internet's most popular zones (the Moz 500 is an easy one to download) through a bash for-loop against "host -t ds".
There is no evidence of any cache-poisoning attacks on DoH servers that I'm aware of. DNS attacks are in general far rarer than you'd think given the attention they've received in the past (again: there isn't much you can do with a DNS cache-poisoning attack other than trying to get a certificate mis-issued, and CAs have multiple levels of defense against that attack).
It's interesting that you played with DNS cache poisoning 25 years ago. So did I! Small world. I wrote a QID-brute-forcing cache poisoner in late 1995 (to get onto IRC from funny domains with). Towards the end of 1996, I wrote the DNS cache poisoning checks for Secure Networks Ballista scanner, which did QID stuff a la Johannes Ulrich, and the DNS authority record caching tricks Kashpureff used, plus some others.
It's fun stuff! There's not much to do with it anymore, though. In 1995, reliable domain forgery would get you past rutils filters and NFS exports; it was a very big deal, so much so that people took the time to do crazy things like elaborate source-routing attacks. But just a year or two later, everything was SSH; a few years after that, and most sensitive things on the web were SSL (it wasn't good SSL, but DNS wasn't the low-hanging fruit).
It's been hard times for people who wanted to spend their careers protecting the DNS.
There are, of course, lots of good reasons not to want to deploy DNSSEC. A good starting point would be that it drastically harms the reliability of all your sites, because when DNSSEC fails, every site in your zone vanishes as if it was never there. That's what happened to HBO the week they rolled out HBO NOW.
But there are strong architectural arguments against it, too.
I end up saying a version of a bunch of this stuff any time people complain about DNSSEC not being available on this or that online service. I appreciate the opportunity to write a new and hopefully more interesting version of the comment. Thanks!
And (I'm ninja-editing this in, so sorry if I caught you while replying), as you noted, much of the growth in .COM signings is attributable to Cloudflare auto-signing --- which is the same thing that is happening in places like .NL. But registrars auto-signing domains is security theater. Ironically: attacks against registrars are far more common than attacks on the DNS protocol itself; the recent spate of "DNS attacks" from the end of last year turned out to have been phishing attacks.
Your same argument seems to imply that since we have good TLS on many protocols, we shouldn't be too heartbroken about people stealing traffic with BGP, either.
And every new deployment of 1990s-grade DNSSEC makes that problem harder.
If we started from scratch, the very first thing that would change would be the notion that cryptography is too expensive to support online signing, which is one of the two original sins of DNSSEC. The entire protocol would be drastically cleaner, easier to deploy, and, of course, much more secure.
This is not the core argument I have against DNSSEC (for that, just Google [against DNSSEC]). But it's a good one. What they say about PHP holds true for DNSSEC: it's a fractal of bad design.
I don't know what BGP has to do with any of this except to say that DNSSEC obviously doesn't do anything to prevent BGP hijacking attacks; attackers who control BGP control IP addresses, the things DNS maps in the first place.
I'm doing my best to keep these responses varied and interesting, because an HN search will show this is an argument I've made here a lot. Hopefully I'm doing an OK job of that.
Root zone has been signed with 2048 bit keys since 2016. Large numbers of domains are signed with 2048 bit keys, too.
> If we started from scratch, the very first thing that would change would be the notion that cryptography is too expensive to support online signing, which is one of the two original sins of DNSSEC.
Online signing still looks really expensive. Asking an authoritative server to do a 2048 bit signature per answer is costly, given the other conversation you've followed me into and argued basically the other side of. Especially since work you do to make DNS authoritative servers verify things is easily used for DOS, unlike work done for prefix verification in BGP).
> don't know what BGP has to do with any of this
Because we're in a thread talking about preventing BGP hijacking of prefixes with crypto, and the tangential discussion of preventing DNS hijacking of names with crypto (DNSSEC) came up. Some arguments you presented against caring about DNS security work equally well wrt: caring about BGP security--- who cares if someone intercepts our traffic, we're probably using TLS!
That the root zone was signed with 1024-bit RSA in 2016 is its own dunk, and I think you for making it for me, but of course the root zone isn't the only zone; the 1024-bit root just meant every TLD was capped at that level of security. Now they're not, but the rest of the DNS is still littered with them. See: every DNSSEC survey.
And, of course, you've addressed only a small part of that previous argument --- just the key sizes.
I don't know what point you're trying to make about BGP. I think you think I'm opposed to RPKI. I'm not.
I agree DNSSEC isn't wonderful, but perfect is the enemy of good. I'd take 768 bit RSA over "doing nothing"-- it raises the difficulty level of a successful DNS attack enormously.
Some of us don't care about enumeration. If we do, there's whitelies, as you say, and eventually NSEC5 will be around.
> I don't know what point you're trying to make about BGP. I think you think I'm opposed to RPKI. I'm not.
You argued earlier that worrying so much about redirection isn't justified: "Say we protect DNS (we could, with a better protocol design). But other redirection protocols aren't protected. Do we similarly authenticate OSPF? IP options? ARP?"
In which case, maybe we should just ignore RPKI, too. :P And unfix source routing, not bother with SEND, etc.
Further evidence on this is easily obtained both from survey papers that collect DNSSEC misconfiguration statistics (they're hilarious) and from the IANIX collection; many of the organizations IANIX notes are extraordinarily clueful operators.
I'm right about this. It's feasible to get DNSSEC deployed globally (it would be a catastrophe, but we indeed have the tools to make that catastrophe happen). But it would be very, very expensive.
This has been a fun thread, but we are way into the right margin now and are probably the only two people reading it, so I suggest we tie it off at whatever your next response is.
IPv6 implements source routing too, but again is hardly used. However it does include an other standard that also works like source routing for mobile networks, but is no where as powerful.