DNS Flag Day on February 1, 2019: check your domains
dnsflagday.net
dnsflagday.net
"This change affects only sites which operate software which is not following published standards. Are you affected?"
Affected _in what way_? Instead of linking to the Wikipedia page about what DNS is (if your visitor doesn't know that, then the rest of the page is pointless), there needs to be a sentence or three up top telling me what is _actually happening_ and what the _actual impact_ could potentially be, and for that matter, which standards you are even talking about. Then, tell me what the forms you are asking me to type my domain names into are going to do.
As it is, this page is just going to leave a lot of people confused.
I did, then I came here to see if you stuck it out. Thanks!
A number of DNS software and service providers have announced that we will all cease implementing DNS resolver workarounds to accommodate DNS authoritative systems that don’t follow the EDNS protocol.
Domains served by DNS servers that are not compliant with the standard will not function reliably after February 1, 2019, and may become unavailable.
If your company’s DNS zones are served by non-compliant servers, your online presence will slowly degrade or disappear as ISPs and other organizations update their resolvers. When you update your own internal DNS resolvers to versions that don’t implement workarounds, some sites and email servers may become unreachable.
Why make this change now?
Extension Mechanisms for DNS were specified in 1999, with a minor update in 2013, establishing the ‘rules of the road’ for responding to queries with EDNS options or flags. Despite this, some implementations continue to violate the rules. DNS software developers have tried to solve the problems with the interoperability of the DNS protocol and especially its EDNS extension (RFC 6891 standard) by various workarounds for non-standard behaviors. This is not unlike the way a driver with the right-of-way might hesitate at an intersection before proceeding if there were another driver in the intersection behaving erratically. These workarounds excessively complicate DNS software and are now also negatively impacting the DNS as a whole.
The most obvious problems caused by these workarounds are slower responses to DNS queries and the difficulty of deploying new DNS protocol features. Some of these new features (e.g. DNS Cookies) would help reduce DDoS attacks based on DNS protocol abuse.
> Minor problems detected!
> This domain is going to work after the 2019 DNS flag day BUT it does not support the latest DNS standards. As a consequence this domain cannot support the latest security features and might be an easier target for network attackers than necessary, and might face other issues later on. We recommend your domain administrator to fix issues listed in the following techical report: redacted.
The minor error reported is that Route 53 does not respond with a "BADVERS" error in response to a query reporting EDNS version 1.
There is no EDNS version 1 standardized, and the specs say to respond with an error when you see a version you don't support. Per the specs Route 53 is "in the wrong". The thinking from the spec authors is that this kind of behavior will make it easier for resolvers to negotiate future EDNS versions and figure out what works. I'm skeptical that this will ever actually work in practice: most protocols tend to have backwards-compatible version upgrades that don't require round-trips.
The thinking on the Route 53 side, or at least my thinking - I wrote Route 53's EDNS support - is that we see incorrect versions occasionally on the wire, from weird clients where people have clearly made some kind of mistake, and that it is better to fail gracefully and give them /an/ answer than an error that could lead to outages. I might be completely wrong about that, but that's the thinking. We tend to always heavily lean in the direction that is likely to increase availability and avoid any potential for outages.
Seriously, if you have responsibility for a large system (ie, think the GE network) - you can take the AWS route - and uptime will be good, or you can do the backwards incompatible changes - and everyone will hate you. Seriously, just the printer DNS lookup clients / the email lookup clients / the amount of cruft in a big system is mind bending.
The other thing I'm not getting, your going to be doing DNS round trips to get to version match? DNS is on the critical path to web page response - I hope this is a joke. Put this into a hint in a recursive lookup or something so when I look up what DNS server handles abc.com I (can optionally) see what EDNS version that server supports? Or TXT field or something? Does the error returned say what is supported? Or will the client need to do multiple R/T's to find the correct version? That's can't be right - I'm not a DNS expert, but I'm not seeing the point of this approach.
None of your rant addressed my question at all - is what Route53 is doing standard practice, or, is it the unusual one?
If you see "incorrect versions occasionally on the wire from weird clients", perhaps the people building those weird clients will know to fix them if they get BADVERS errors.
The other side of this is that being too tolerant can lead to network ossification. This impacted TLS1.3's roll-out, which had to be made to look similar enough to TLS1.2 session resumption that many network middle boxes can't tell the difference and let the traffic through. This is less elegant than a cleaner new design that isn't tainted by having to appear like the old formats.
Personally, I worry about this a lot less and consider the extra effort to be backwards compatible very worth it. It think it will still be very easy to find totally unused never-seen-in-the-wild EDNS version numbers; there's no shortage. On the TLS working group we've been through this a few times, having to find unused magic numbers that hadn't been burned as part of the various experiments. We've done with this protocol versions, cipher suites, etc.
On the DNS side; the proposed hypothetical EDNS version negotiation scheme that might show up in the future requires round-trips and state. Resolvers would have to first send an EDNSv1 query, observe it fail, store that state somewhere and then try with EDNSv0. What if the authoritative server rolls back their support? What if the auth service is mixed and only some of the fleet supports the new version? These challenges tend to make that kind of version negotiation impractical ... hence my skepticism that it will ever work like that.
A small part of all this is that the DNS WG isn't really stewarded with these practical issues front of mind. DNAME and DNSSEC have each had backwards-incompatible implications that real-world operators had to fudge around and ignore precisely what the specs say, to be able to satisfy customers. Due to that history, IMO the DNS specs have lost a certain level of moral authority that other specifications still enjoy.
That's completely understandable.
What I'm wondering is whether you could accept it for now, but reach out to customers you see it from and provide guidance, in addition to blogging about the issue.
I'm not suggesting "turn it off tomorrow and break people", I'm suggesting "work towards a long-term plan of turning it off".
I assume (hopefully) that Route 53 is going to upgrade eventually.
https://forums.aws.amazon.com/message.jspa?messageID=851817#...
Quote for those without access: There is a known bug with Route 53 DNS servers in which we are not RFC compliant in how we handle a specific kind of invalid query. Namely, when Route 53 gets a query with an unknown EDNS version, Route 53 treats the query as a non-EDNS query instead of responding with BADVERS as ednscomp expects.
We expect to have this bug fixed within a year or two. The good news is that this bug is not impacting, so you'll be ok even if we're slow to fix this. The "dnsflagday" news means that servers that don't support EDNS will be treated as unavailable to resolvers. But Route 53 generally supports EDNS0, valid queries will continue to work regardless.
So it basically means that authoritative DNS servers are now required to support the EDNS protocol. If not, it is no longer guaranteed the domain will resolve on DNS resolvers.
This is a performance improvement, because the EDNS fallback method requires a timeout.
Edit: My answer isn't correct, see the comments below.
EDNS0 ameliorated this greatly, allowing clients and servers to keep talking DNS/UDP without falling back to DNS/TCP, up to much larger packet sizes. That is primarily why one would want it, even if one did not want any of the other things that it incorporates.
Also, a lot of DNS hosts do not allow for large packet sizes over UDP to attempt to reduce the effect of reflection attacks.
Do any allow a large response if you send a large (padded) request?
So, yes, I think padding it could work. (EDNS does have a padding type)
"The use of the EDNS(0) padding only provides a benefit when DNS packets are not transported in cleartext. Further, it is possible that EDNS(0) padding may make DNS amplification attacks easier. Therefore, implementations MUST NOT use this option if the DNS transport is not encrypted."
Apparently it does have a padding proposal, but it wasn't thought through very well. They only had the use case of confidentiality in mind, and decided to deal with amplification by forbidding cleartext use, no matter what the response:request size ratio is.
It basically needs to be the final record, which conflicts with things like SIG0 that also want to be the final record.
edns=ok edns1=noerror,badversion,soa
ednsopt=ok edns1opt=noerror,badversion,soa
Wikipedia doesn't mention anything about EDNS versions, so I'm not sure what to make of it.https://github.com/PowerDNS/pdns/issues/6806
This issue isn't affected by the configuration change on flag day and judging by the slow adoption so far, by the time where edns support really gets mandatory, the current version of the distro at that time will long have updated their built-in PowerDNS version.
And about time, too.
The EDNS fallback method does not require a timeout. A standards-compliant DNS server that does not support EDNS should ignore EDNS records in the request and simply send an old-style response that any EDNS capable resolver is still required to handle correctly. The problem is only with servers that simply don't respond to requests containing EDNS records at all, even though they are required to by the (old, pre-EDNS) standard.
To most people, it's not the "domains" that should be checked, but DNS servers and DNS resolvers, both the authoritative and recursive type. If you are using a major DNS provider for your domain, no action is needed, but just to be sure, use the test tool on the webpage to see if your provider has broken EDNS, and do check your local recursive server.
Classic DNS messages carried by UDP were restricted to 512 bytes, EDNS boosted this restriction and also introduced some flags, and it has been enabled by major DNS servers since 1999. But in practice, many deployments on the authoritative servers are broken, they signal EDNS support, but EDNS replies are silently dropped, due to broken DNS servers, misconfigured router, broken NAT, broken ISP installations, or broken firewalls or other middleboxes.
Previously, various DNS resolvers contained a workaround that disables EDNS as reaction if a DNS query timeout is detected. Now the workaround will be removed. If a DNS resolvers has EDNS but it's broken, it will be marked as a dead server.
Thanks HN commenters for doing the site's job :)
If you're running a DNS server and have problems, it's probably not a simple matter of blowing out the latest version of your software. You probably have a rat's nest of thousands of entries added over a decade or more, different firewall policies in front of different authoritative servers, caches, etc.
I am very familiar with DNS. I run the tests and get minor failures. The test results barely make any sense and don't really offer good pointers on how to resolve them.
The documentation is obtuse in typical ISC fashion.
I could probably figure it out with more effort. But the average DNS Joe won't, they'll ignore this and move on.
> Serious problem detected!
> This domain will face issues after the 2019 DNS flag day. It will work in practice, BUT clients will experience delays when accessing this domain. We recommend you request a fix from your domain administrator!
Weird since they are listed as one of the supporters
So I can imagine after 20 year it is time to move on.
When I first read it I thought it was some kind of 'new' tech.
This isn't so much "You must offer a vegetarian option" as "Don't offer a vegetarian option but then when anybody orders it you force a steak down their throat while screaming 'Choke on it you hippie!'".
The industry's complacency in getting broad EDNS support has been a huge embarrassment imnsho. This is proof that we don't need to put up with "but the middle boxes, argh!!?" excuses if the industry works together to force people to be compatible with standards.
Thanks to all those involved with making this project a reality. It really helps everyone on the internet, whether they know it or not.
They don't list it, but presumably the same applies to the Authoritative servers. I run a pretty vanilla NSD install and it fails several tests. NLNETLABS (the authors/maintainers of nsd/unbound) is listed as a supporter, but they don't mention anything about this on their site nor does their documentation mention anything about settings to control these features.
If this will actually have the consequences they describe it was terribly mismanaged.
I think the point is that those versions mentioned will remove support for the bad behaviour, but it's not clear what you do need to do to prepare. I guess the only steps you can take are to run the tests and update to the latest version available.
Apparently the fix is to use the external repository from powerdns.
I kinda get the need for a web frontend for a C program (portability is an issue), I miss the point of the second front-end. Why?
I understand the technical problem but I'm not convinced this is as big a deal as it seems. There are some big names associated with the flag day event, lending credibility. But there are also big names like Slack.com, Twitter.com, GitHub.com, BitBucket.com, and web hosting companies, that all fail the EDNS/dig test.
So - the stated threat is that sites will go dark on 1 Feb, but do we really expect that to happen? Might these companies simply be threatening to reject invalid EDNS responses? Will people blame Them for rejecting queries rather than the services being resolved and their DNS?
I can't decide if this is something to scream loudly about, or to just let it happen. I'm not asking for opinions on that. I'm asking for a credible source of information to confirm what is truly expected.
Thanks.
> Minor problems detected!
> This domain is going to work after the 2019 DNS flag day BUT it does not support the latest DNS standards. As a consequence this domain cannot support the latest security features and might be an easier target for network attackers than necessary, and might face other issues later on. We recommend your domain administrator to fix issues listed in the following technical report [link to report]
As other comments have said, the new rule does not require that all domains support EDNS, merely that all domains respond with “EDNS not supported” when applicable instead of pretending that they support it. Those sites you linked follow that rule.
See Jargon File.
The big ones like Mark Monitor, etc are super expensive. I thought Cloudflare Business DNS would be cheaper, but they start at 1000-2000$ per year.
Any recommendations ? I have heard of Safenames..but not many others.
It's free, except for the official signature verification, that can be done at the post office for $1.5.
The home page mentions that EDNS queries will be honored even for non-DNSSEC queries.
It does not exhibit any of the faulty behaviours that are common to bad EDNS0 implementations. It does not blindly echo various parts of the query back in the response, for example. Indeed, the ISC test actually complains that tinydns does not blindly echo an OPT record back. (-:
But you cannot speak DNS/UDP to it with datagrams over 0.5KiB. Patches are needed to make it do anything but wholly ignore EDNS (of any version) and treat it as though it were the unextended protocol. But that is not what this flag day is about.
Indeed, given that tinydns does not send DNS/UDP responses greater than 0.5KiB, it does not trigger any EDNS0 problems in the network infrastructure that a server is using, either.
Ironically, one of the people who tried to rewrite djbdns to have EDNS0 support discovered exactly the problem that this flag day is about: The bad content DNS servers for outlook.com in 2017-2018 gave bad responses to clients that attempted to speak EDNS0.
Needless to say, after tons of time getting into the weeds with EDNS, and STILL not understanding why certain clients weren't working, that was a fun one to solve.
I don't know of any latest-version DNS software that breaks in this case. AFAIK this is exclusively a middleware problem, as is quite often the case.
What is happening is that all of the stuff that I have been explaining in the "Local fix" section for a decade and a half, alongside the practice that clients had of timing out and then retrying the transaction without the extensions, is now being declared moot. If DNS lookup does not work because of this problem, it is being declared entirely the server-end's problem. Clients are no longer going to be expected to put such local fixes in place, or have timeout and retry bodges.
If you only run hosting at netlify (thus, having your DNS CNAME to Netlify) you can still have problems.
Use the checker on the website to know for sure.