The Sisyphean Task of DNS Client Config on Linux | Hacker News Reader

The Sisyphean Task of DNS Client Config on Linux

tailscale.com

167 pointssmitop85 comments

Tailscale is awesome, you should use it for everything.

This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.internal" and "rabbitmq-cluster.internal".

What you'd want in a perfect work is an option in `/etc/resolv.conf` that sends `.internal` to a special nameserver, and everything else to a normal nameserver. But as far as I can tell, there's no way to take a bare Linux VM that can accept an arbitrary container and set that capability up.

So instead, we end up (by default; you could override) serving _all_ customer DNS, and our `.internal` server has to forward recursive queries to things that aren't `.internal` somewhere else. This sucks; we shouldn't have to be inline for arbitrary customer DNS.

If there's a clean way to resolve this, so that the VM itself can just send `.internal` queries to us, and everything else to `1.1.1.1` or `8.8.8.8` or whatever the customer's container had, I would _love_ to hear it. I've come pretty close to breaking out preload in anger over this problem.

Couldn't you run dnsmasq, unbound, or some other configurable resolver on the localhost? Though that's probably already what's done with the internal smart resolver.

This is proper advice. My recursor of choice is pdns-recursor , but bind in forward-only mode with several forwarders for different zones will work too

dnsmasq doesn't do authority checks so unfortunately an external cname to a .internal name may not resolve correctly if the upstream server gives back "additional data" containing the A record answer for the target .internal name. On the other hand, we use unbound instead of dnsmasq to solve this at work and it works great. If you just want to forward internal vs everything else differently it's great though.

If you run CoreDNS instead, they have a kubernetes mode of operation where it gives different answers based on the source IP of the query to optimize search domain lookups. So if the node hosting the VMs here did similar, you could solve this pretty easily using CoreDNS with a custom plugin but I don't love putting all my traffic through CoreDNS vs a more mature DNS server.

bind can do split horizon stuff based on source IP so that could be a potential solution, though I'm not sure if it's capable of split horizon forwarders.

systemd-resolved can do this, though I haven't played with it too much yet.

Here's a random post I just found: https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b...

(Ha, I did the thing where I read the comments before the post, and the post describes how to do this. So what's still missing?)

I'm not sure how reasonable it is for us to run systemd-resolved (or systemd itself) on our VMs. We're trying to provide a clean environment for any random container to run on; we provide our own init, and that's almost the whole of it.

What, I think, you really care about here is glibc's behavior. But then, you can't depend on any one libc, either.

The cliffs of cloud native software defined networking insanity await you:

https://github.com/containernetworking/cni

https://github.com/firecracker-microvm/firecracker-go-sdk

  Firecracker, by design, only supports Linux tap devices. The SDK provides facilities to:

  Attach a pre-created tap device, optionally with static IP configuration, to the VM. This is referred to as a "static network interface".
  Create a tap device via CNI plugins, which will then be attached to the VM automatically by the SDK. This is referred to as a "CNI-configured network interface"

It's unreasonable to bring the whole cow to just get a gallon of milk . Use standalone resolver.

Funnily enough, that is the _exact_ same problem I'm facing right now, down to Firecracker and a custom internal TLD. I'm excited to see the solutions. I think the only difference is that I need to run this DNS service on the same host as my VMs, so I will need to use a different port than systemd-resolve.

I'd recommend to run it on the same port, but different IP (127.0.1.2 , f.e.) , due to inability of some programs to use non-standard port

So apparently systemd-resolve does support redirecting custom TLDs to a specific nameserver [0], as mentioned by dnr. However, support for custom ports was only added in version 246 and my Ubuntu 18.04 ships with 237.

As far as I can tell, my way forward is to use v246, then set my TLD to my custom nameserver with my custom port and add a stub listener on localhost (instead of 127.0.0.53, which is the default). Then I can offer my VMs to use their tap gateway (which is the Ubuntu host) as a DNS server if they want, and that will resolve my custom TLD and forward all other stuff to its own nameserver. Or, if the user wants, they can also do their own stuff.

It seems that a lot has happened with systemd-resolve in the last few months.

Feels like the actual clean way to resolve this is to not use DNS for discovery of named services.

On macOS, you could do this trivially by creating /etc/resolver/internal with the name servers in it.

Linux should copy that.

Systemd-resolved does much more than MacOS resolver. It can make the additional resolvers active conditional on whether the link is up, for example (think VPN). MacOS resolver can't do that.

I thought tptacek was looking for something simpler than systemd-resolved. I was just pointing out the simplicity of /etc/resolver/<domain> on macOS that's been available since, I think, the first version of macOS. It's a nice feature.

In any case, on macOS, you use scutil to adjust resolvers when you change networking state as when bringing up a VPN.

Conceptually they are similar; one has config files in /etc/resolver/*, other in /etc/systemd/resolved.conf.d/*.conf; one has mDNSResponder process (yes, it is also unicast DNS resolver and cache, since 10.6, despite the name) and the other systemd-resolved. With systemd-resolved, you can achieve exactly the same thing as with MacOS resolver, for approximately the same cost.

With systemd-resolved, you don't have to adjust resolvers manually as links come up and down, NetworkManager's VPN connections play nice with it, while DNS resolver handling in Mac VPN software (including Wireguard and Tunnelblick) is a nuasance :(.

... I realize I'm a monster, but aren't y'all already extensively using BPF for things? Sticking in a BPF egress filter on the VM that rewrites outbound DNS packets with .internal in the question to point at your DNS server seems like it would be lighter weight than just handling all queries recursively.

Why do you need kernel-mode parser for that ? iptables -t nat -I POSTROUTING -p udp --dport 53 -j DNAT --to <your internal recursor, sending .internal to authoritative dns server for that zone, and resolving globally available hostnames by itself>

Well, because he doesn't want to be inline for non-.internal DNS queries.

You can technically add an iptables match rule to only forward DNS packets whose contents match a ".internal" DNS query, but it sounds like a recipe for disaster. It would be better if they wrote an actual iptables protocol filter for DNS (assuming one doesn't exist) but that's so much work for so little benefit.

sure, but that's why BPF is so great; I'd guess a reasonable program that assumes port 53, parses the packet enough to look at the first question and compare domain against .internal and then rewrites address + updates checksum is maybe... 300-400 LOC? can probably get a bit fancier, but it shouldn't be too painful to write and will execute plenty fast enough.

This is good praxis. I have received some thoughts about how to do this without parsing DNS in eBPF (which would be a disaster).

The main problem with this is that you need to handle TCP connections, in which the queries do not have to match up to packet boundaries, and the client can multiplex internal and external queries on the same connection.

The interface for DNS from an arbitrary container is the DNS protocol, and since you can’t control the software that is making the queries on the client side of the protocol, you have to do the special logic on the server side.

Do you, realistically? The standard behavior for most things afaik on looking up a domain name is to send out an A query and (simultaneously ideally) A queries with searchdomains appended. That's not gonna get anywhere near 512 bytes for a fly .internal service name, I'd expect.

The requirement is to support arbitrary containers, so it should still work if the container contains stub resolver software that pipelines requests over TCP.

It depends on what you're using for the resolver. I'm assuming you only care about gethostbyname(3) and friends. With glibc that means nss; generally you're also looking at libnss_dns.so, which uses glibc's resolv (copied from BIND). This doesn't include enough configuration to do what you suggest; it pretty much just points everything towards a server.

So you have two options: use a different NSS module (maybe write your own?) or have a proxy DNS resolver that sends different requests to different places.

systemd-resolved actually handles the first option pretty well (although it would prefer that you use the dbus interface over gai). It can handle multiple interfaces with separate domains and split DNS fairly well! (Not so good with reverse DNS, unfortunately. But I get it, reverse DNS is pretty hacky anyways.)

If you prefer the forwarder route, dnsmasq seems to be fairly popular these days in the embedded world and elsewhere.

If I were you, I think I'd write a short NSS module or use dnsmasq, depending upon your needs.

Indeed, dnsmasq seems to be the least painful way to provide special DNS zones, special host files, and forwarding for everything else.

Also, your resolv.conf becomes trivial.

The trick is to grab the DHCP-provided DNS server address on reconnections and update the forwarding, if you use a laptop. For VMs in the cloud, it's not a problem, of course.

I used to have a very similar problem (didn’t want my company vpn to handle all dns traffic, only what was directed at the company), I managed to solve it with a local dnsmasq on my machine.

Unlikely to fit your usecase. But it’s not impossible for us plebs.

If you're willing to assume glibc, a reasonably clean approach is to write your own nss_fly module that returns results for .internal.

The downside is that Go code will drop to cgo for all resolutions (because it will see something it can't handle in pure Go in resolv.conf) and non-glibc code (like Alpine containers using musl, or non-libc resolvers like ares) won't get anywhere at all.

But you're kind of doomed in that latter case, anyway, because since there currently isn't a standard for how resolv.conf should express the rules you want, the effort to get it adopted by glibc, musl, Go, Chrome, ares, and all the other various interpreters of resolv.conf will take way too long to make a practical impact on a startup's product.

I can think of a variety of ways. Some involve running some code on the VM. For example, dnscache or dqcache would work. This can also be complished using tinydns. Tiny programs that use very little memory. But it sounds like you are trying to avoid asking the user to install anything other than flyctl on the "bare Linux VM". What is not clear is what programs are installed by default in the "bare Linux VM". The other question is how many ".internal" domains the VM will need to resolve. The simplest solution that comes to mind is when the user provisions an IP, flyctl writes the ".internal" domain(s) for that IP to /etc/hosts. /etc/resolv.conf can then point to whatever the user prefers. IMO, users should be running their own DNS servers, not setting /etc/resolv.conf to point to third party DNS addresses like 1.1.1.1 or 8.8.8.8 or whatever. In the case they are running their own DNS server on the VM, then it becomes trivial segregate internal from external domains. The configuration for dnscache is so easy that it could be done by the flyctl program, requiring no user interaction. (Something like tinydns-config.) One could even have configurations for a variety of DNS servers in flyctl, in case the user prefers unbound, etc.

If it is flyctl doing the resolution then another idea is to have flyctl do the resolution in a chroot with its own resolv.conf.

> This sucks; we shouldn't have to be inline for arbitrary customer DNS.

Why not? Someone has to do it, and if I were paying Fly for service, I'd rather have Fly handle my DNS queries than be another freeloader on Google or Cloudflare.

This is also what we're trying to do in Tailscale (to grab your MagicDNS domain, and whatever corporate split DNS you set, but not anything else). And yeah, on linux, basically systemd-resolved is the only thing that gets this right (with dynamic config - with fully static config, other recursors are an option, with varying tradeoffs), everything else assumes "one resolver should be enough for everything".

So, your solution to proxy all the traffic and split out as needed is the right way to do it without adding more software into each VM :(

Doing stuff via LD_PRELOAD would be hilarious, you should definitely do that and report back ducks behind the blast shield

You're trying to do something DNS wasn't built for, so, there is no good solution for what you want ("only serve .internal and nothing else"). As I see it, you have 3 options:

1. Use ".internal.someotherdomain.io" and the regular old public DNS. (this probably only does half of what you want)

2. Use dnsmasq at the edge of each .internal subnet, and maybe require the customer to configure your DNS in their container. (I believe this is what AWS does?)

3. Intercept tcp/udp port 53 at the subnet edge, spoof your own responses for .internal, pass through anything else to wherever it's going. Dealing with tcp could be tricky, udp is easy. Probably this will break some clients, and generally be the hardest thing to implement.

I think the best option is #2, as it will actually work reliably and with the least difficulty for everyone.

(2) is what we do, modulo that we don't run dnsmasq. (We slip our DNS configuration in via our init).

I'm not persuaded by appeals to what DNS was or wasn't built for (unfortunately neither is the IETF; to wit: DNSSEC). I agree with you that the ask here is big! But I'm not convinced there isn't a cleaner solution that works within our constraints.

Then another option is to implement all 3. This gives you what you want most of the time, but also supports all customer use cases. The downside is the customer now needs a flow chart to understand how their DNS is resolving.

To start with, first implement (1).

Assuming .internal points to internal-only IPs, and you are reusing internal IPs (each customer VPC can have arbitrary private IPs), you can have ni-172-16-0-5.customer.internal.somedomain.io return A record to "172.16.0.5". (You don't have to add .customer, but you might gain some useful analytics)

A customer needing a hostname for 172.16.0.5 can resolve ni-172-16-0-5.customer.internal.somedomain.io. By default it'll use whatever DNS server/resolver they have configured (or yours with option (2)). The request goes through public DNS until it gets back to somedomain.io (your domain) and you return the result.

Upside: You're not hosting all the customer's DNS resolution. Downside: The customer will see a longer round-trip for .internal requests, and their VPC needs a NAT gateway.

Next implement (3).

Intercept all DNS traffic that matches ".internal.somedomain.io" and spoof responses. Write this tight enough that it will only return responses when you are sure they won't break clients, and pass the rest to the public DNS using option (1), which prevents from having to mess with TCP connections (they'll be slower though the public internet, but it's TCP, what did they expect?).

Upside: You are only returning results for .internal, everything else can use somebody else's DNS, and the customer doesn't need a NAT gateway. Downside: Requires custom engineering which is subtly complex, and is hidden magic the customer doesn't expect, so could cause troubleshooting headaches.

Next, implement (2) as an opt-in.

Have your VPC DHCP server return your internal dns server. You resolve .internal locally and handle the rest as a big hulking resolver. You can also slip this DNS server into their containers, but as a customer I'd want a giant red flag in the docs that advertises this.

Upside: Works in every customer scenario. Bonus: now you can return reverse records. Downside: Now you really are hosting the customer's DNS resolver.

> If there's a clean way to resolve this, so that the VM itself can just send `.internal` queries to us, and everything else to `1.1.1.1` or `8.8.8.8` or whatever the customer's container had, I would _love_ to hear it. I've come pretty close to breaking out preload in anger over this problem.

If you're at that point, rather than preload, why not simply add the code to glibc and contribute it back? Has there been some resistance from the maintainers already?

It would seem like it should be doable to add it on in a safe way, e.g. something that would work for both: `nameserver 192.168.0.1`

and ``` nameserver 192.168.0.1 nameserver 10.10.10.1 .internal .private .etc ```

Even if there's hesitance in accepting a change to the `nameserver` options, you easily make this an option, e.g.: `options tld-nameserver:.internal:<addr>`

This just adds more complexity to a universally used user space library. And also breaks when some programs don't use the same resolver. Just implement your own dns server that does it for you (or reuse dnsmasq).

I guess you've heard of dnsmasq which I'm pretty sure can do all you want.

Perhaps it doesn't count as bare.

[EDIT: not only do many other mention this, but the article does too! my bad]

Yes, we built our own DNS server in Rust, we're aware of dnsmasq. The problem we'd like to resolve is not having to have a SPOF DNS server we're responsible for in between our users VMs and the Internet.

The prospect of running dnsmasq on every VM we launch is to aversive to consider. At that point, we'd just put a DNS recursor in our init.

Multiple redundant servers? I'm way out of my depth here, but if a spof is your main problem, perhaps you just need some redundancy?

But shouldn't this be solved with Multicast DNS [1] and DNS based service discovery [2]?

Sounds to me that your .internal TLD is causing a lot of headache and sticking to avahi and .local would be easier?

[1] https://tools.ietf.org/html/rfc6762

[2] http://dns-sd.org

What we deploy has to work with any reasonable application; we build to the lowest common denominator interface.

People on this thread have a lot of ideas about what they'd do if they got to be the sysadmin for every container running on our fleet.

How about generating a custom resolv.conf per instance, protecting it from systemd via `chattr +i`, then run any other services you need to inside?

This is called "Split Horizon DNS". The problem is, there is no standard way to communicate this information to a machine. e.g., afaik there is no DHCP extension to say "for .internal, please see this server, for the rest, see whatever". So you pretty much have to run a server in the middle.

It -is- possible, as others have commented, to do this within the VM or container but they need a way to get that information. systemd-resolved does this. From what I understand of fly.io, the container is setup by the user, so you're not going to have control over their networking configuration or environment - is that right?

Now there is an alternative, which is instead of using .internal, to use a public facing internet hosting that is delegated to your internal DNS servers. That way they use their normal DNS, and it resolves to your nameserver. But the DNS needs to be public, and critically, if the internet DNS goes down, your .internal resolution will also stop working. Which is highly likely to cause problems for peoples applications. And also adds un-predictable latency to their local service connections which is almost certainly going to cause problems - it's not a good idea.

Having said all of that, generally speaking, it makes sense that -you- provide the DNS. It's pretty much the standard way that your 'ISP', 'Hosting Provider', etc, provides the DNS rather than relying on the public internet DNS. Only a few years ago there was no such public DNS.

The other possible alternative would be to leverage multicast DNS (libnss-mdns/avahi-daemon) or LLMNR but I'm not sure those are really ideal for this use case either. And in both cases requires resolver support that you dont get from the basic glibc resolver. So for your use case, may still not be workable.

So I think you're stuck, unless you can mandate that all of your VM images use systemd-resolved and somehow communicate the config into them.

We do and we don't control the user environment. We can run anything we want in a container, and we can tamper with the runtime environment of a container entrypoint any way we want. But what we come up with has to be compatible with any container someone throws at us.

Additionally: the basic problem of routing DNS requests based on domains is trivial. It's probably less than 100 lines of Rust to send `.internal` to our servers and anything else to a public recursor. But the goal is to get our own code out of the path of people's DNS lookups, even if it's just 100 lines of Rust.

So running dnsmasq or some systemd resolver server thingy doesn't really solve anything for us. Before we did that, we'd just make our `init` a DNS server. But it'd be neat to come up with a way to to solve the problem without having a SPOF server dependency. It might could be possible! You can already see that if you assume a specific libc, you can just preload a libc stub resolver that routes requests. But, of course, we can't assume a particular libc.

Most of my point here is just that "standard" Linux DNS is bad.

Bad compared to what? Is there any OS that has a better DNS resolution process/methodology? Windows? BSD? Solaris? Serious question. I mean do they all suck or?

The most interesting takeaway from this article is that, according to people who actually do the work, NetworkManager and systemd-resolved do get things right.

one reason resolv.conf is/was so sticky is that gethostbyname (and friends) is a libc thing and pretty low-level

Problem is, modern browsers do things differently yet again. So the insanity continues, just on different levels.

browsers are operating systems in user space. They will continue to replace kernel parts.

Technically, the existing dns resolving mechanism is a part of glibc, so it is userspace already.

Mostly right. Article never mentions /etc/hosts , which is still largely a thing, and works wonders in difficult cases ( and makes other difficult cases much worse to debug)

Because /etc/hosts is not really part of the chain handled by resolver or nss-dns. It is handled by different nss module which usually has higher priority.

I agree! As a non-expert, I've also had good luck with NetworkManager, but wouldn't recommend systemd-resolved yet if you use DNSSEC:

https://github.com/systemd/systemd/issues/6490 (fixed in v248)

https://github.com/systemd/systemd/issues/8451

https://github.com/systemd/systemd/issues/9867

https://github.com/systemd/systemd/issues/12388

To be honest I hate all of the aforementioned programs trying to battle over /etc/resvolv.conf. That's why by default I have the file marked as immutable, and pointing to 127.0.0.1. This way I cannot have accidental DNS traffic leaking. If I need local network DNS I send a manual dhcp command to get the wifi's DNS and add it temporarily to my dnscrypt config. Similar thing for VPN DNS.

I do something similar whereby I just point DNS traffic at the router itself and configure it per-host by DNAT, as part of the larger WAN horizon zone (eg direct goes to local recursor, wireguard tunnel goes to the recursor on the other side of the tunnel or suitable public resolver, commercial VPN goes to whatever their recommended resolver is, eventually pihole for guest network). Running multiple apps / security contexts / nyms (or whatever else you want to call them) on the same OS instance is just asking for trouble.

Wait wait wait. You forgot the part where if using systemd-resolved you are supposed to symlink /etc/resolv.conf to /run/systemd/resolve/resolve.conf, which gets generated after boot. Also should caveat that you will no longer be able to resolve / recursive search based on only a hostname, services using systemd-resolved instead of glibc will need to provide fqdn. Real pain in the ass.

I recommend the series "Anatomy of a Linux DNS lookup"[0].

Everything you never wanted to know.

https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-l...

That appears to have been written prior to systemd-resolved becoming enabled by default on many distros.

Which systemd is still unable to do DNS split horizon with DNSSEC.

- ISC Bind9 is still king.

I also tried once to make sense of domain name resolution: https://venam.nixers.net/blog/unix/2020/11/01/resolving-a-ho...

If you are just trying to use curl or some other straightforward small application - this is enough. But running Kubernetes, docker, or some other container system? This mess is much more deep...

For example: there are known issues with systemd-resolved & Kubernetes (at least on Ubuntu that defaults to systemd-resolved) https://kubernetes.io/docs/tasks/administer-cluster/dns-debu...

FYI, if you don't bother opening the link, systemd-resolv sets the /etc/resolv.conf to have only 127.0.0.1 as a DNS server. You can see how things get bad when the CoreDNS pod tries to get the "upstream" DNS servers from the host /etc/resolv.conf

I guess it shouldn't be doing that? I mean, that also breaks in the case where you're running something like dnsmasq as your local resolver, which will also require 127.0.0.1 in your resolv.conf.

As much as I'm not always comfortable with the larger complexity, the CoreDNS pod really needs to be doing something like the flowchart described in the article. If systemd-resolved or NetworkManager are in play, it should be using D-Bus to talk to them to get the information it needs.

resolv.conf is an old idea that is too inflexible for today's DNS needs; a more complex solution with more complex interface is unfortunately warranted here.

> I mean, that also breaks in the case where you're running something like dnsmasq as your local resolver, which will also require 127.0.0.1 in your resolv.conf.

Dnsmasq DNS support is redundant when you are running systemd-resolved. It can do everything dnsmasq does and more.

> resolv.conf is an old idea that is too inflexible for today's DNS needs; a more complex solution with more complex interface is unfortunately warranted here.

That's exactly what systemd-resolved is.

It also messes up docker-compose, probably in the same way.

I have to disable systemd-resolved and manually configure resolv.conf to get my containers to resolve each other properly. (Also Ubuntu)

I’m confused, this article seems to assert that systemd-resolved can handle split DNS based on the domain name and even claims the docs for it are fantastic. But last time I went down this rabbit hole I finally gave up and combined it with dnsmasq. The docs are verbose but useless, and I ultimately determined it was impossible. systemd-resolved would failover to alternate DNS servers but you’d end up making a ton of failed requests. So if I’m wrong and you can actually send requests to DNS servers based on their TLD please tell me how.

It depends on how long ago you checked, but it can do it now without dnsmasq.

https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b...

An enlightening article that explains why I think I am losing my mind every time I try to get the network configured correctly on any number of distros. My main take home is to never install NetworkManager or a resolvconf and to avoid systemd if at all possible, unless I absolutely know that I am going to need those state of the art DNS capabilities.

I have mostly managed to avoid resolv.conf issues since the default Gentoo image has a sane setup (even if using dhcp on a laptop and switching wifi and wired). However, every single time I have tried to set up a system using another distro something has gone wrong, and in support of the theory that NetworkManager is a major contributor to the problem the one Gentoo system with NetworkManager installed had the same issues.

To echo the plea in the original article, in nearly every case the primary challenge has been to figure out exactly what documentation actually applies to the system at hand because distros seem to change this completely out of sync with any attempt to correct or align the documentation.

I suspect that on an individual user level this leads to a happy path situation where everyone who does the right thing by accident is quiet and the ones who need something slightly different are never heard from because they weren't able to even connect to the internet (probably not quiet that bad).

That's exactly the wrong take home; the only sane way to handle dns is systemd-resolved, as the article said. If you use your machine as anything resebling desktop/laptop, you should configure your network using Network Manager, especially if you have connections that come and go. Switching between Lan and wifi with dhcp is really low bar.

Anything else is just prolonging the agony that the linux networking configuration had to endure for years.

Where does /etc/nsswitch.conf factor in? I don't see it mentioned.

It’s probably omitted as nsswitch doesn’t affect DNS client configuration.

It affects which sources are consulted to lookup a name (including maybe asking DNS), but it doesn’t configure things like which DNS servers to ask or what options to set.

> The Name Service Switch (NSS) configuration file, /etc/nsswitch.conf, is used by the GNU C Library and certain other applications to determine the sources from which to obtain name-service information in a range of categories, and in what order.

nsswitch.conf is what tells it whether to use the traditional nss_dns (the thing that looks at /etc/resolv.conf), or whether to use the newer nss_resolve (the thing that talks to systemd-resolved).

So yeah, if the article is contrasting resolv.conf vs systemd-resolved, nsswitch.conf is how you select which of those approaches is used.

The article focussed on how /etc/resolv.conf is being managed and populated. That’s not affected by nsswitch.

I only addressed why nsswitch was likely not mentioned, not that it doesn’t serve a purpose.

Even with nss_resolve, you still have to have /etc/resolv.conf correctly populated, because some apps will ignore gethostbyname() and just parse resolv.conf and do DNS by themselves. For example, golang stdlib does that (but to the credit of golang runtime, it checks whether nsswitch.conf is in expected state and falls back to glibc if it is not).

Some apps go even further and do their DNS entirely on their own (Firefox DoH controversy).

nsswitch however doesn't just configure mechanism for public resolver, it is a hook where alternate mechanisms like nss-mdns or nss-mymachines can hook up.

nsswitch.conf tells it whether to consult `/etc/hosts` (`hosts: files`), `/etc/resolv.conf` (`hosts: dns`), or systemd-resolved (`hosts: resolve`); and if multiple of those, in what order (`hosts: first second third...`).

Ok, now let's talk about how messed up the same thing is in Mac OS.

Hint: it is waaay worse

Dave, one of the authors of these posts, has been regularly documenting on Twitter his struggles with DNS on all the platforms.

If you like sports, it is like watching a basketball game, with three teams and Dave as the play by play announcer.

I'm a linux user, so I'm always cheering when OSX or Windows "loses" but then very quickly Dave will, paraphrasing say: "And, OSX hits a deep 3 pointer..." ("And, OSX does this right, and Linux sucks").

But, it is great theater and very informative. On Dave's recommendation, I switched to systemd-resolved, and it is working flawlessly for me (with my own wireguard setup, not using Tailscale yet).

It's a pity, because macOS got the general idea right, but seemingly every single particular wrong thereafter.

In general, a modern DNS client wants: a set of "default route" resolvers; a set of "DNS routes" that point certain suffixes to other resolver configs; a set of search paths to expand single-label queries; integration with mdns and LLMNR, for seamless zero-config resolution on LANs (super important for printers, in particular); all of the above tied to interface lifetimes, so you can tie resolver reachability to underlying network state; very detailed documentation on the algorithm used to resolve a name, and how you traverse all the above configuration.

macOS has default resolvers, DNS routes, mdns integration (but no LLMNR), interface-tied configs, and knows about search paths.

But then you look at the NetworkExtension API for configuring DNS, and it turns out the search paths field doesn't actually configure the search paths in ways you'd expect, instead all the suffixes you install as "routes" end up also becoming search paths, and your only option is have all or none of them be search paths. Meanwhile, the search paths you specified do get installed... In an interface-scoped config that doesn't actually get used in the majority of name lookups that need name expansion.

It's so frustrating because it's this close to being excellent, and instead ends up being the most limiting of APIs we have to work with, because being apple, it's either their API or go screw yourself and don't configure DNS.

Oh dear, I've ranted again, haven't I. Anyway, every OS is its own beautiful little snowflake of weirdery an brokenness. Linux's particular flavor is "there's 15 ways to do it, most of which require polyfills". macOS's flavor is "we have an API that should be amazing but somehow does the wrong thing almost always". Windows's flavor is "we can do really cool things but the main source of documentation is people exchanging superstitions about registry keys on stack overflow".

Given that choice, I think I prefer linux. It's way more code to write to make it work, but at least the code can be derived from documentation+source code, and has half a chance of working as desired.

Wait till you see what happens when Zscaler gets added into the mix for corporate laptops.

I now have to scenario for when Zscaler intercepts/rewrites DNS requests along with when the Cisco VPN is active. Given that a large amount of folks are still WFH, it’s enough to make me want to tear the remainder of my hair out.

Can't agree more. They have 'clever' way doing a lot of things, which make us suffering to troubleshooting

But wait, there's more! nsswitch.conf and nscd.conf also affect the stub resolver's behavior.

I've been avoiding figuring this out years and resorting to editing /etc/resolv.conf manually and chattr +i it to make it readonly. I'm a NetworkManager user so guess it's time to set it up through there

systemd-resolved still doesn’t work with split-horizon DNS with DNSSEC on both sides of the horizon.

Reply on news.ycombinator.com