Hacker News Reader

> Route 53 is a so-called "child centric" DNS server, in that it considers the child zone to be fully-authoritative for its own NS records. That's why the name servers for a parent zone reply with a referral, rather than in the answer, because from their perspective, they don't really "know" the answer. The proper thing a resolver should do is follow the referral and ask the child name servers.

> I don't think that's in violation of what the DNS RFCs specify

Agreed, and in fact it's pretty easy to see that the root and .org name servers behave exactly like Route 53 in this regard.

> In any event, I think what's going in this case is more subtle but simpler, basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should, but that takes a very convoluted edge-case set up involving two over-lapping delegations from the same parent zone.

Yes, that appears to be what is happening, and it breaks regular DNS resolution, not just QNAME minimization. If your DNS resolver has the NS records for ws.fdmg.org cached because you previously looked up a ws.fdmg.org hostname, you won't be able to resolve any prod.ws.fdmg.org hostnames, because your resolver will send the queries to the name servers for ws.fdmg.org. I don't know if this is a Route 53 bug or if the customer misconfigured their zones, but if the latter it would be nice if Route 53 could stop people from shooting themselves in the foot this way. To reiterate, this breaks regular DNS resolution, not just QNAME minimization.

That said, Route 53 improperly returns NXDOMAIN for empty non-terminals, which does break QNAME minimization (and violates RFC8020). For a proper example, see opsmate0._domainkey.sslmate.com and _domainkey.sslmate.com.

RFC8020 is an example of the DNS WG making backwards incompatible changes. The "NXDOMAIN" for an empty non-terminal has long been the behavior in DJB's TinyDNS.

That isn't what Route 53 is using, but I bring it up because there have been mailing list threads (or flames?) where folks complained about what DJB's software is doing, and he repeatedly asked "Show me where in the standards this is is specified or forbidden?" and no-one would come up with the goods, but folks still basically wanted "Just do what bind does".

Here's DJB's own mail about it, I had to forward it because his mails don't get through to the mailing list for some reason (that might be the fault of his own stupid qsecretary moderation system): https://www.ietf.org/mail-archive/web/dnsext/current/msg1108...

I generally sympathize with DJB in this case, because DNS really needs some robust and competing implementations. The standards should be tolerant of differences in behavior (as others are), rather than dominated by the proponents of one implementation. It doesn't help that the WG is toxic, at least in my opinion as someone who has been reading it (and mostly just reading) for over a decade. Anyway, I'm not trying to start an argument, just explain. In this case, the bind proponents did dominate, and we get RFC8020, which for some software is a backwards incompatible change.

This isn't the first backwards-incompatible change either. The invention of DNAME introduced a new kind of backwards-incompatible occlusion, and name servers "before" and "after" DNAME respond in different ways, for the same zone file.

Those kinds of changes place providers in a pickle: one of our golden rules is never to make backward incompatible changes or break anything that was working. A promise is a promise. So it means we have to lock things down (the previous paragraph is the reason Route 53 does not accept arbitrary literally-encoded RRs) and when changes come; either not make those changes, or exhaustively audit and determine how many zones, queries it might impact and whether that would change anything. At scale, you'd be amazed how small and subtle behaviors can end up being embedded as assumptions in customer scripts and so on.

I'm with you. I read the complaint twice, and the RFC twice, and I don't quite think it's a bug. It's a weird thing to reason out, but I don't think Amazon's behavior here is wrong, though it might be unexpected...

For various values of unexpected.

> The proper thing a resolver should do is follow the referral and ask the child name servers.

You're right, I was clearly not paying attention when writing that part down (fixed now in the forum post).

> [...] but I also strongly suspect that QNAME minimization may rest on under-specified or un-specified ideas about what authoritative name servers should do. I'd have it follow referrals if it doesn't already.

AFAIK, that is not the case, QNAME minimisation is well-specified, and the resolver will follow referrals if it gets them. But in this case the resolver is getting an NXDOMAIN where it should be getting either a NODATA response (if there is no zone at ws.fdmg.org), or it should be getting a referral, if there is a zone at ws.fdmg.org, which should then contain a delegation for prod.ws.fdmg.org.

Since it's getting neither, the resolver will stop resolution when it gets the NXDOMAIN. Now it may be the case that this is either broken empty non-terminal behaviour, or there is actually a zone at ws.fdmg.org, but that zone is missing a delegation for prod.ws.fdmg.org. It's hard to tell from the outside, but if the latter is the case it should be easy to fix.

There is a zone cut at ws.fdmg.org, so I think it's the latter. The zone is misconfigured, that's pretty simple to fix for whoever owns the zone, but the general question of whether Route 53 should be more strict and prevent it from being imported or not is a tough call.

On the one hand it's good to prevent these edge cases and keep things deterministic, but on the other it can cause availability issues for customers if Route 53 were to refuse to import a zone that probably loaded and "worked" just fine with the previous or secondary set up. There's all sorts of ways it's possible to write a mis-configured but "valid" zone file, where records and subtrees of records become unreachable. Most customers rely on test queries and traffic monitoring to observe if there are problems.

In general, at AWS we try to avoid being paternalistic; we believe in giving customers powerful tools and then making those tools intuitive and easy, but I'll admit this can be very challenging with DNS, where the standards and obscure corner cases were never really intended to be understood by an inexpert user-base.

I resisted making this point last night because I didn't want to be seen to deflect. But on reflection, the privacy advocate in me really needs to get this out: QNAME minimization is a really really bad idea and doesn't do anything meaningful for privacy IMO.

The most privacy sensitive domains are still leaked. DNS queries are still in the clear, including the ultimate specific queries for the domain you want to resolve, observable to anyone who can look at your traffic, maybe including your wifi buddies at the local café.

Then along the way the sensitive names are still leaked to parties you probably shouldn't trust in the privacy-protecting model. For example: Verisign can still tell that you are querying for "oddlyspecificpornfetish.com". Dropping the "www", if there even is one, isn't much protection. In return for essentially zero meaningful privacy protection, you have to deal with all sorts of name resolution problems due to making NS queries. That's not a smart trade-off. I'm not saying the goal isn't worthy though, it is.

Agreed, it is a tough call on whether or not Route 53 should try to prevent this type of misconfiguration. I would err on the side of saying you probably shouldn't because before you know it you're going down the rabbit hole of fixing all sorts of weird misconfigurations users make in the DNS.

And yes. DNS seems so simple on the outside until you let a non-expert near it. And even experts make mistakes analysing corner cases (mea culpa).

As to whether or not QNAME minimisation offers privacy, this is debatable, but here I would err on the side of saying: it's a building block that at least prevents some leakage (to e.g. the root).

A more important takeaway for me here is that QNAME minimisation makes misconfigurations such as missing delegations actually cause problems, whereas this would have been masked if QNAME minimisation is not enabled. I'm planning to do a measurement of the frequency at which this occurs.

Finally, as agwa pointed out, Route 53 still has broken empty non-terminal responses, I've added a separate reply to the AWS forum thread with the example agwa used.

> On the one hand it's good to prevent these edge cases and keep things deterministic, but on the other it can cause availability issues for customers if Route 53 were to refuse to import a zone that probably loaded and "worked" just fine with the previous or secondary set up. There's all sorts of ways it's possible to write a mis-configured but "valid" zone file, where records and subtrees of records become unreachable. Most customers rely on test queries and traffic monitoring to observe if there are problems.

Since this problem only materializes if you prime your resolver cache correctly, it's really easy for testing/monitoring to miss it.

Route 53 could issue a warning instead of rejecting the zone outright. For example, Google Cloud DNS issues these warnings if you try to add an SPF TXT record without quotes:

"Warning: A record for this domain has whitespace but is not a "quoted string" and therefore is split into separate strings at whitespace. SPF, DKIM, and DMARC join those strings without spaces, which can cause "problems, especially for Sender Policy Framework records. Warning: A record for this domain starts with "v=spf1" but lacks a quoted space following the '1'. This may be a badly formatted Sender Policy Framework record that will be ignored by mail software."

That saved someone I know from publishing a bad SPF record.

Thanks for the insights, always interesting to read things like this.

Is there any chance that you can add this to your internal "customer want this feature"?

I like to be able to add multiply hard/software second factor devices to on IAM Account.