Hacker News Reader

Comment by wrs | Hacker News Reader

These people have quite simply configured their DNS wrong--all DNS servers are expected to return the same results, so it can't matter which server the resolver uses. But their misconfiguration happens to work most of the time for their use case. The solution is either to implement a "bug-compatible" priority flag in resolved, or to make that a first-class use case with its own flag ("emergency backup nameserver"?).

It's a classic case...should the software follow the spec, or the real-world usage? And people complain either way.

View on HN

Real-world usage, obviously. If you're going to replace a foundational component of the OS, you need to replace it in a bug-compatible way, period. No compromises.

A common use for this behavior is VPNs: you sign onto a VPN, and the VPN's DNS server gets prepended to the list in resolv.conf, which allows you to resolve internal addresses (and still fall back to your regular DNS). This is a crazy-common use case, and it's incredibly arrogant of Poettering to just dismiss it out of hand, or even attempt to argue with it.

Which real world usage? dnsmasq and Windows both follow the systemd behavior by default.

Our VPN is configured to replace the server list, which is the correct behavior, because the new servers don't return the same answers as the old ones. So again, those other VPNs are misconfigured.

On macOS, there's a straightforward solution for the VPN use case that doesn't involve a "fall back": the VPN registers its DNS servers for specific domain names, and so anything in example.com has its list of DNS servers (tunneled through the VPN) and everything else has the original list of DNS servers from your physical network connection. See resolver(5) https://developer.apple.com/legacy/library/documentation/Dar... for a description of doing this with files named like "/etc/resolver/example.com" (and having the same syntax as resolv.conf), but you can also do this with SystemConfiguration (googling "SupplementalMatchDomains" should be helpful).

It looks like systemd-resolved implements something similar: https://www.freedesktop.org/software/systemd/man/systemd.net...

The fact that there is an incredibly common use case that consists of an unreliable workaround for people who don't have a local resolver that does this correctly is hardly reason to avoid writing a local resolver that works correctly.

If you have an incredibly simple internal network, that might work fine. It certainly would not work at my organization.

> all DNS servers are expected to return the same result

Really? My local DNS server should return the same results as 8.8.8.8? Should one never use unregistered suffixes?

All listed proxy DNS servers should provide the same view of the DNS namespace. If your local proxy DNS server provides a different view to Google Public DNS, then it is wrong to list them together.

* http://jdebp.eu./FGA/dns-client-all-proxies-must-provide-sam...

And no, you shouldn't just blithely assume that you own parts of the DNS namespace that you do not, either. There are plenty of other people's mistakes to learn from here. Don't repeat them.

* http://jdebp.eu./FGA/dns-use-domain-names-that-you-own.html

I think the argument is that if you do put both your local DNS server and 8.8.8.8 in your /etc/resolv.conf, then yes, they should return the same results. Put another way, a configuration file that enumerates DNS servers isn't the appropriate place to "merge" things.

I don't have a strong position on that argument. Just trying to state it more clearly.

If you limit requests to the different servers to the domains they cover, that's fine (various resolvers do this in different ways). If you tell the resolver they're all equivalent, they need to all be equivalent.

Understood; thanks for clarifying.

No, one should probably never use unregistered suffixes. But that’s not to say that all DNS names should be public, nor that NAT is practically avoidable, so this is beside the point.

What one should do is only choose DNS servers capable of properly resolving all addresses, as it shouldn’t be the resolver’s responsibility to “shop around” for answers.

Fixing VPN configurations that complicate this should be the responsibility of VPN client software — by providing a forwarding DNS server that falls back to the originally configured servers, say — not the resolver, and similarly for any other cases where standard resolver behavior does not suffice.

Application software, for its part, shouldn’t generally rely on implementation-defined resolver behavior, as doing so is obviously not portable, but that’s less sinful than a VPN relying on client library-level support to work around the fact that it’s exposing a broken network configuration.

Note that I have no axe to grind one way or another in the systemd debate: Linux isn’t me primary OS, but I regularly use and abuse both systemd and non-systemd Linux systems, and all the Linux init systems seem to work at least as well as any other system I regularly use (OS X, FreeBSD, Windows) for most practical purposes.

Aside: You are using "resolver" to describe the wrong thing. RFC 1034 employs "resolver" to describe the entire subsystem that user programs use to access what it calls "name servers". It theoretically allows, because it isn't intended to be Unix-centric, systems where all of the query resolution mechanism is potentially in some form of shared subsystem that runs inside each applications program.

Strictly speaking, what you are talking about here, in RFC 1034 terminology, is a "stub resolver", the Unix model where there's a fairly dumb DNS client library running in the applications, talking to an external program that actually does the grunt work of query resolution by making back-end transactions and building up the front-end response from them.

A "resolver" actually very much does "shop around for answers", amongst content DNS servers. There's even a (slightly faulty) description of its shopping around algorithm, in RFC 1034 section 5.3.3.

"resolver" is such a confusing term, and so often used contrary to the RFCs in the way that you have here, that years ago I took to explaining the DNS to people using terminology borrowed from HTTP: proxy servers, content servers, and client libraries linked into applications.

* http://jdebp.eu./FGA/dns-server-roles.html

Well, certainly you should never use unregistered suffixes because how else are you going to get SSL certificates for your internal services?

Self signing, or if your feeling fancy issue from your own CA and import the root cert on your clients.

If they're internal services the namespace and the trust heirarchy can be private just fine.

Self-signing is a great way to invite phishing attacks against your end users: if they're used to clicking through cert warnings all the time, they won't notice when something has actually gone wrong.

Running your own CA works but has two caveats: first, I don't really trust myself to run a CA competently given that almost nobody who runs a CA as their only business function is doing so competently, and second, you have to get your config onto every device on the network. If you let personal devices on the network or even just hard-to-configure things like Chromecasts, you have to get your config onto all of them somehow.

It's quite a bit simpler and more secure to just use .corp.example.com or whatever.

>Self-signing is a great way to invite phishing attacks against your end users: if they're used to clicking through cert warnings all the time, they won't notice when something has actually gone wrong.

Sure, it's most suitable for a homelab where you can import the certs, with suitable discretion, for yourself only.

>Running your own CA works but has two caveats: first, I don't really trust myself to run a CA competently given that almost nobody who runs a CA as their only business function is doing so competently

Not a massive concern since the context is internal only, but I hear you.

>and second, you have to get your config onto every device on the network.

Yep, this can be tricky in a less controlled environment. I had two scenarios in my mind - Home lab, and enterprise, when I penned that response. There's definitely a third one here, that you have in mind, that I didn't cover.

>It's quite a bit simpler and more secure to just use .corp.example.com or whatever.

Definitely the best option for small to medium companies.

> I don't really trust myself to run a CA competently...

You don't have to deal with any of the issues of public CAs because you control all the machines, running one in a corporate environment is no issue at all and there are plenty of existing products that will do almost all of the work for you.

> you have to get your config onto every device on the network

Which is a single task in an Ansible playbook, Puppet module, or Task in the Windows Task Sequence.

> you have to get your config onto all of them somehow

Usually not, you just need to get your CA on all the devices that are going to connect to your hosted services.

No. What is meant here is that whenever multiple DNS resolvers are configured on a particular DNS client, those resolvers should share an identical view of the DNS landscape for that client to consume.

If you have an internal DNS resolver which handles some private DNS entries, you should not set up any client with that resolver AND a second resolver which does not also have those entries.

So what the OP it's saying is that the example you provided is something you should never do.

So to clarify the statement: "all DNS resolvers defined on a given client are expected to return the same results"

No, this is not a misconfiguration. It is a configuration that expects the resolver to behave according to the algorithm described in resolver(5). systemd chose to wedge itself in as a resolver, then broke that contract.

I was curious about this, since it sort of impacts my wourld, and decided to check this. There's a RES_ROTATE flag in there, described as "Causes round-robin selection of name servers from among those listed. This has the effect of spreading the query load among all listed servers, rather than having all clients try the first listed server first every time." Claim checks out.

resolver(5) describes the behavior of glibc, not DNS or systemd. Systemd will read resolv.conf if you don't want to configure resolved.conf directly, but that's a convenience, not a contract. Dnsmasq also reads resolv.conf for convenience, and puts yet another interpretation on it (which will also break this use case). Both systemd and dnsmasq expect that you'll tell glibc to just ask the local resolver (them).

Not all DNS servers will return the result in the same amount of time and same amount of network/computing resources.

If the ordder is predictable, I can do some shadowing: if A is searched before B, I can put some entry into A which blocks certain sites or redirects.

Even if we just have one server A, there is still an order! The recursive DNS search implements that order. Each server in the chain has the option to return its own local result, which doesn't have to coincide with the next server that is used.

So, that is to say, DNS is implicitly ordered from the leaf server you're using up to the root one.

The resolving proxy server that one sends queries to, from a client library (built into the C library or otherwise), is not a "leaf". This is the same erroneous mental model of the DNS that got the world the completely wrongheaded tool known as dnstracer, that bamboozled people for years.

* http://jdebp.eu./FGA/dnstracer-incorrect-algorithm.html

There is not a "chain" of servers, each passing on queries to the next one along. There is a resolving proxy DNS server that takes a front-end query, makes a bunch of back-end query+response transactions with content DNS servers, and stiches their results together to make the final front-end response.

* http://jdebp.eu./FGA/dns-query-resolution.html

The aforementioned DNS client library can be given the IP addresses of multiple such resolving proxy DNS servers, but there has not been one single conventional order of querying them for far longer than systemd has been around. Different DNS client libraries do it in different ways. nslookup's internal DNS client library does something different to the ISC DNS client library that forms part of many C libraries, as I have already pointed out in this discussion.

The whole "chain of servers" model is one of the commonest misunderstandings of the DNS, in my personal fairly long experience of people with DNS problems. It is quite wrong.

The pre-systemd behavior is that the order is not really completely predictable.

First of all, DNS queries are usually made with UDP. There is no ack or retransmission with UDP, and the network stack is free to drop messages if necessary, such as if a queue fills up or the network is congested. (Hence it has the nickname "Unreliable Datagram Protocol".)

Second, "man resolv.conf" has this to say about what happens when you have multiple DNS servers listed: "The algorithm used is to try a name server, and if the query times out, try the next, until out of name servers, then repeat trying all the name servers until a maximum number of retries are made."

Consider what happens when you combine those two together. You roll the dice and send a UDP request to the first server. Then it rolls the dice by sending a UDP response to you. If that doesn't work, you send to the second server. If that doesn't work, then you try the first server again.

TLDR: In the case of lost packets, which you must assume happen because it's UDP, it alternates between servers. The only thing that's guaranteed is which one comes up in the rotation first.

> These people have quite simply configured their DNS wrong--all DNS servers are expected to return the same results, so it can't matter which server the resolver uses.

So much for CDNs or internal networks then.

> all DNS servers [in resolv.conf] are expected to return the same results

Better?

Yes, actually :) That's a crucial distinction.

> all DNS servers are expected to return the same results

Is that assumption true anymore, given a world where some ISPs issue redirection for unfindable domain name[1]?

[1] http://www.dslreports.com/shownews/103386

I think it's, perhaps, a faulty assumption, but a correct expectation, and ISPs that issue redirection for unresolvable domains have in fact misconfigured their DNS servers.

EDIT: Also, Comcast shut down their Domain Helper as it is fundamentally incompatible with DNSSEC

http://corporate.comcast.com/comcast-voices/comcast-domain-h...

I hadn't heard about that. Thanks for the info.

Reply on news.ycombinator.com