DNS glitch that threatened Internet stability fixed; cause remains unclear | Hacker News Reader

DNS glitch that threatened Internet stability fixed; cause remains unclear

arstechnica.com

98 pointsdangle132 comments

Cogent's statement on c.root-servers.org:

"2024‑05‑23 - On May 21 at 15:30 UTC the c-root team at Cogent Communications was informed that the root zone as served by c-root had ceased to track changes from the root zone publication server after May 18. Analysis showed this to have been caused by an unrelated routing policy change whose side effect was to silence the relevant monitoring systems. No production DNS queries went unanswered by c-root as a result of this outage, and the only impact was on root zone freshness. Root zone freshness as served by c-root was fully restored on May 22 at 16:00 UTC."

Edit to add:

This was mentioned on Tuesday on the dns-operations list:

https://lists.dns-oarc.net/pipermail/dns-operations/2024-May...

The problem was evident if you ran

dig @${server} . soa

c.root-servers.net showed an older serial number than all the others yesterday.

This seems like a major hole in their monitoring systems.

Internet stability is not threatened by the loss of a single root server, it’s not even a blip on the radar. In fact most people wouldn’t even notice if they all went down for a few minutes, DNS operators do not actually use these servers directly. You can even run your own if you are crazy enough.

Everyone likes to meme on DNS but it’s literally designed to make these servers as unimportant as possible. It’s a really well designed system that will probably be around as long as humans exist.

> DNS operators do not actually use these servers directly.

No, but a lot of servers do. In 2020 the root servers saw 120 billion queries a day, which was double what they saw in 2018:

https://blog.apnic.net/2020/09/28/scaling-the-root-of-the-dn...

It's probably closer to half a trillion queries a day now.

IIRC a large chunk of those is spam from chrome which most people wouldn't notice if they stopped working

… under what circumstance is Chrome hitting the root DNS servers?

Old bug fixed in 2020. OP didn’t know of the fix it seems. Was covered on Hacker News at the time. Still top google result.

Well that's horrifying.

For those that are wondering like me: https://blog.apnic.net/2020/08/21/chromiums-impact-on-root-d...

Essentially, it generated 3 random, single-label (so, "TLD") queries, and ran them through DNS. Whatever recurser that lands at obviously wouldn't have random junk in its cache (unlike probably any other root record) and thus had to go to the root servers for it.

They did this to try to not display an omnibox result for single words (the example in the article is "marketing") on what I'm going to just call "broken networks"; see TFA if you're curious.

Note that running your own mirror of the root on the recursive nameserver(s) on your network will keep the traffic generated from these sorts of queries off the rest of the internet.

If you're already running your own recursive resolver on reasonably up-to-date software it can be surprisingly easy to set this up; see rfc8806 for some config examples.

Is there a use case for that? I mean other than that it's kinda cool

Per rfc8806, because the data in the root zone has long TTL values, most queries to the root servers are for nonexistent top-level domains due to typos and search paths gone wrong, so in some sense it's community service to reduce useless traffic to the roots -- you do a single burst download of about 2MB daily instead of all of your query traffic.

Other reasons why you might do it are covered in rfc8806 - slightly reduced latency on cold-cache DNS queries, added resilience against DoS attacks against root DNS servers.

> that will probably be around as long as humans exist.

My goodness, that's a strong claim to make, about a glue layer in a computing technology that's been around for just 50 years. I mean, arguably the internet as we knew it is already dying. The original design of DNS was flawed (no security), and the patches to fix those flaws are far from perfect.

Replacing Internet infrastructure is incredibly difficult. We'll all be switching over to IPv6... any decade now.

DNS was flawed, but it was very good for the time, and it still is the keystone around which the entire Internet functions. That's pretty incredible for a system that has indeed received a lot of patches. And DNSSEC might turn out to be just as well designed as the original DNS. Besides, what are we going to replace it with, namechain? I'll put you on hold while all the ISPs switch over.

I think the conventional wisdom is that we are going to replace DNSSEC with nothing. Security is provided at other layers of the stack.

Certain forms of security cannot be provided by other layers of the stack.

> but it was very good for the time

It's still very good. People keep finding new ways to apply DNS.

I was just grumbling about the OTT claim that the DNS will still be there as the last human becomes extinct.

Over 50% of Internet traffic is IPv6, by the way.

A bit pessimistic, but maybe the subtext is we won’t be around for very much longer

it's Good Enough and Not Broke, so there's little incentive to iterate

I wouldn't mind some alien technology taking over for TCP/IP but that's squarely within the "unforseeable" IMO

Not surprising behavior from Cogent, who to this day refuses to peer with Hurricane Electric's IPv6 network [see https://en.wikipedia.org/wiki/Hurricane_Electric and related footnotes].

If it takes 3 days to be noticed by an outside party, I dare say that whatever caused the issue probably isn't the first problem to fix.

Honestly the issue here isn't that they let the zone get stale. The root zone doesn't change that much. So a couple of TLDs had to hold off on making changes, who cares.

The issue here is that they didn't notice until someone with a direct contact at Cogent emailed them to tell them.

A corollary to that issue is the fact that the last news item is the same thing: they failed to get updates and their monitoring failed to tell them. Supposedly their monitoring was fixed then, in 2019. Why should we believe it's any more fixed now?

Cogent sucks. Their routing issues are so common and often severe. I seriously don't know how that org still exists.

Here is one glitch that seems to be related that still is not fixed:

https://www.indilib.org/forum/general/14691-indilib-org-site...

The glitch began on the 18th and is now in the situation where some users can and others cannot access the site. I, myself, cannot access the site six days later, when connected via my network wifi router (ISP is Cox Communications), but can, if instead I connect to the Hotspot on my IPhone(AT&T). Traceroutes fail in either case, continually, at the server 198.207.200.70 (which does not appear to be owned by cogent).

What, if anything, can an end-user such as myself do in such a situation. Where can such an end-user send a report that might lead to a fix? I assume the owners of the indilib.org site are trying whatever they can, but I have no idea what their reporting chain might be.

What's your evidence that this is related to c.root-servers.net serving a stale version of the root zone?

If you can get the correct IP address for a site out of the DNS but can't reach the site, it's not a DNS problem, and not a problem with the root DNS servers. This is likely correlated only in time with the c.root-servers.net issue.

That said, your difficulty reaching indilib.org may be related to the Cogent-TATA peering dispute or one of Cogent's other peering disputes - from where I sit, a traceroute to indilib.org goes through cogent's network. But that's not what this submission is about.

Probably accidentially firewalled the DNS-XFER/IXFR port for a few days.

I know this similar scenario happened to our lab setup of root servers.

Don’t tell Andy Jenkinson about this.

The biggest issue here is a total lack of monitoring of root zone sync among all 1840+ root server instance, where an issue is detected accidentally by a random dude.

Huge oversight from ICAAN and ARIN. I know they are behind anycast, but each server must also have an unicast address for management.

clickbait headline

Ehh, from TFA:

"As changes accumulate in the root zone, the difference between the versions begin to matter more. Typical changes in the root zone are changing delegations (NS records), rotating DS records for DNSSEC updates and with all this comes updates to DNSSEC signatures. If the differences remain, then the outdated root server will see the DNSSEC signatures expire and that will begin to have more marked consequences."

I mean, the specific "more marked consequences" are unspecified, but I think "could affect stability" is a reasonable interpretation thereof, and I do personally care that they don't know wtf caused it yet, so...

But Cogent do know what happened. The article ends with a quote from them:

> On May 21 at 15:30 UTC the c-root team at Cogent Communications was informed that the root zone as served by c-root had ceased to track changes from the root zone publication server after May 18. Analysis showed this to have been caused by an unrelated routing policy change whose side effect was to silence the relevant monitoring systems. No production DNS queries went unanswered by c-root as a result of this outage, and the only impact was on root zone freshness. Root zone freshness as served by c-root was fully restored on May 22 at 16:00 UTC.

Ahh, ty, I should have read more completely/carefully. Thank you for the correction.

Reply on news.ycombinator.com