Anatomy of a Linux DNS Lookup | Hacker News Reader

Anatomy of a Linux DNS Lookup

zwischenzugs.com

168 pointsbelter49 comments

Follow up:

Anatomy of a Linux DNS Lookup – Part II https://zwischenzugs.com/2018/06/18/anatomy-of-a-linux-dns-l...

Anatomy of a Linux DNS Lookup – Part III https://zwischenzugs.com/2018/07/06/anatomy-of-a-linux-dns-l...

Anatomy of a Linux DNS Lookup – Part IV https://zwischenzugs.com/2018/08/06/anatomy-of-a-linux-dns-l...

Anatomy of a Linux DNS Lookup – Part V – Two Debug Nightmares https://zwischenzugs.com/2018/09/13/anatomy-of-a-linux-dns-l...

This does not cover the joy of eg. connecting to multiple networks (different DNS settings) with separate interfaces.

Then you can hit DNS request hopping between servers too: https://github.com/systemd/systemd/issues/19209

Before networkmanager and systemd, pretty much every tool used gethostbyname, which used standard nsswitch which tended to be hosts, then resolv.conf

resolv.conf would use the first one, and then timeout and use the next one

Nice and simple.

The quote

"simply turn off systemd-resolved"

Is usually the answer, but it's a constant uphill battle to fix broken behaviour.

Of course Pottering doesn't care.

And straight away it's a downvote.

Pottering, in that thread, literally says

"I have the suspicion that this going to negatively affect about as many people as it positively affects..."

https://askubuntu.com/questions/907246/how-to-disable-system... is a "good" solution to the problem in ubuntu

To be honest, systemd-resolved actually for the first time provides an option of a solution for this use-case. Properly configured, without CNAMEs involved, it would just do the right thing.

Relying on resolv.conf meant using the first one unless it times out, which makes it impossible for me to connect to two private networks with their own hostname resolution, if a hostname resolves in both but to different addresses.

But how can you do that unless you are saying "map this .foo.com to resolver A, and map .bar.com to resolver B".

You have an option to install dnsmasq or whatever resolver you want for that very unusual use case without installing tons of more bloated demons on machines that simply don't need it. 99% of linux machines are not laptops connecting to multiple networks with private DNS servers, they have static network configurations, or a single network connection with dhcp. Of the rest, most are laptops, and sure you want some form of gui selection to choose your wireless network, and maybe even set up your vpns, but that's such a rare case.

Also remember that private DNS servers are on the way out now on a desktop, with DoH being embedded at the browser layer, so this tiny use case will shrink even more.

by all means have a "yum install dns-multipathd" or whatever as a resolver to solve this problem on the 1 in a thousand machines it helps on, but why have any software installed and listening on port 53 by default? Seems a needless security risk to me

> But how can you do that unless you are saying "map this .foo.com to resolver A, and map .bar.com to resolver B".

That's exactly what systemd-resolved does for you when properly configured. The linked issue is about something.foo.com resolving to blah.bar.com and then that hitting resolver B, and Poettering correctly notes how it might cause issues for some while it fixes issues for others.

Now, whether I think systemd and the ecosystem is too complicated? Sure it is, and that means it's a guaranteed pile of bugs. But I also believe the goals are admirable, and it is up to everyone to decide whether they are willing to live through the bugs (thus helping improve it) or not, relying on more traditional tools.

I am not sure how DoH is going to interact with local network DNS as I haven't looked into it, but if browsers are unable to figure it out the right way, I'll for sure disable it.

So that's nothing to do with the network you're connected to, but do do with things like search domains.

The unix way has always to be provide an optional daemon (like dnsmasq, other daemons are available) to do unusual setups like this. It's not to do with networking, it's a higher level than that. I might wand .google.com to resolve via 8.8.8.8 but normal domains to resolve via 1.1.1.1 -- both of which might be via the same gateway, or perhaps via a different gateway on the same subnet, or perhaps a different interface entirely. Maybe I want .blah.com to connect via DoT but others to not. Perhaps my routing protocol changes the gateway I use to reach the servers I want.

None of that should be of any concern to the DNS forwarder

DHCP will provide a nameserver and search domain for the network you connect to, which will resolve domains in that search domain properly. There is nothing special about that.

With systemd-resolved, the right thing happens out of the box. Without it, you've got to do something arbitrary like you describe.

It's ok to dislike systemd (there are plenty reasons to do so), but if you won't acknowledge things it does well, it's hard to take you seriously.

FWIW, I don't think having a local network (with eg. a NAS drive on it) and connecting to another (VPN) is so uncommon.

> DHCP will provide a nameserver and search domain for the network you connect to

No, it may provide a nameserevr which I may or may not want to use.

> which will resolve domains in that search domain properly

It may, or may not, provide a search domain. Chances are I'll be using that domain server for all domains, or I'll choose to use my preferred server (4222, 9999 etc)

My vpn on my laptop doesn't even use dhcp so not sure how that would apply.

> FWIW, I don't think having a local network (with eg. a NAS drive on it) and connecting to another (VPN) is so uncommon.

Bridging networks, generally not a good thing.

That's not bridging networks: you are instead connected to multiple networks simultaneously, whereas in bridging, you make one network over multiple interfaces.

As for your configuration, it seems you are connecting to a bunch of misconfigured networks if you want to use your own prefered servers.

"Disable networkmanager, uninstall network manager, delete networkmanager, nuke networkmanager from orbit" was always my 4 step process for making things work right for ages. (and all 4 steps were always somehow necessary, because it would always come back from the dead).

I think you have the pieces all jumbled. Every tool still uses gethostbyname which uses the glibc NSS machinery. The only thing systemd did was add a few NSS modules that could (and are) configured in /etc/nsswitch.conf. You probably have the 'resolve' and 'myhostname' modules. The module that reads /etc/resolv.conf is the 'dns' module.

The behavior of the 'dns' module is terrible. If you have three servers listed and the first one is unavailable then every. single. dns. request. must wait for the first server to time out then try the second one.

Also if you have three servers listed and server 3 happens to be the fastest you'll never know because even if the first two servers are down and you finally use it there are two timeouts in between you and that speed.

And since there's only one resolv.conf if you're connected to a split-tunnel VPN you get to choose whether to send all your DNS requests over the tunnel or none of them.

systemd-resolved actually does the right thing in (almost) all of these cases. Built-in kernel IPSec still trips it up because systemd-resolved's view of the world is mapping DNS configuration to interfaces and since kernel IPSec is a policy VPN blah blah nevermind -- it has memory so it can remember if servers are down, and it used to send requests to all listed servers to find the fastest one and prefer it before people complained, and split-tunnel VPNs work!

The "feature" of "merging" DNS zones by relying on query order is pants on head broken as soon as your first DNS server is unreliable or if your first server doesn't return a strict superset of all subsequent servers. If you have internal names both over your VPN and on your local network then no matter what order you pick one of them will be unresolveable.

And if I want to install a forwarder like dnsmasq or resolvd I've always been able to do that. A lot of the problem that people have with systemd is the forcing of it by the distros.

Not to be overlooked on this topic is this wonderful Tailscale blog entry:

https://tailscale.com/blog/sisyphean-dns-client-linux/

and corresponding HN: https://news.ycombinator.com/item?id=26821298

  * disclaimer: I have no affiliation w/ Tailscale. I just wish I did.

This predates systemd's decision to get involved via systemd-resolved. So now it's got another step :)

Edit: Well, and also browsers doing DNS over https. It's pretty confusing these days to know which path your app took to resolve a name.

Which is super annoying when you actually want to run a DNS server and have to try and convince systemd-resolved to stay in its own lane.

Is there a good example of a Linux package that does this correctly?

I suspect the big part is putting "DNSStubListener=no" into the [Resolve] section of /etc/systemd/resolved.conf if you want to have something else listening on port 53.

Or you can just mask the service (better than disabling):

    systemctl mask systemd-resolved.service

Other services to consider masking if you are not using SystemD for network configuration and time sync., etc.:

    systemctl mask systemd-hostnamed.service
    systemctl mask systemd-timesyncd.service
    systemctl mask systemd-networkd.service
    systemctl mask systemd-homed.service

Thanks; got a writeup abt these you might be willing to share?

Yeah, but if you regress to 'legacy DNS' by removing systemd-resolved then there's no good way to do per-interface DNS (~client-split DNS), or (optionally) validate DNSSEC, or do DoH/DoT; and then nothing respawns and logs consistently-timestamped process events of substitute network service processes.

FWIU, per-user DNS configs are still elusive. Per-user DNS would make it easier to use family-safe DNS (that redirects to family-safe e.g. SafeSearch domains) by default; some forums are essential for system administration.

Since nothing in the real world is or ever will be DNSSEC signed anyways, it's not that much of a loss. Meanwhile, the most important DoH user --- your browser --- has DoH baked in and doesn't need systemd-resolved's help.

Your system may also depend upon one or more package managers that do all depend upon DNS (and hopefully e.g. DNSSEC and DoH/DoT)

Package managers don't rely on DNSSEC for package security (it would be deeply problematic if they did). But you can also just go look and see that, for instance, Ubuntu.com isn't signed, nor are most of the Pacman mirrors for Arch, nor is alpinelinux.org, &c. DNSSEC is simply not a factor in package management security.

People have weird ideas about how this stuff works.

Unfortunately all packages aren't necessarily signed either; "Which package managers require packages to be cryptographically signed?" is similar to "Which DNS clients can operate DNS resolvers that require DNSSEC signatures on DNS records to validate against the distributed trust anchors?".

FWIW, `delv pkg.mirror.server.org` is how you can check DNSSEC:

  man systemd-resolved # nmcli
  man delv
  man dnssec-trust-anchors.d

  delv pkg.mirror.server.org

Sigstore is a free and open Linux Foundation service for asset signatures: https://sigstore.dev/what_is_sigstore/

The TUF Overview explains some of the risks of asset signature systems; key compromise, there's one key for everything that we all share and can't log the revocation of in a CT (Certificate Transparency) log distributed like a DLT, https://theupdateframework.io/overview/

Certificate Transparency: https://en.wikipedia.org/wiki/Certificate_Transparency

Yeah, there's a channel to secure there at that layer of the software supply chain as well.

"PEP 480 -- Surviving a Compromise of PyPI: End-to-end signing of packages" (2014-) https://www.python.org/dev/peps/pep-0480/

> Proposed is an extension to PEP 458 that adds support for end-to-end signing and the maximum security model. End-to-end signing allows both PyPI and developers to sign for the distributions that are downloaded by clients. The minimum security model proposed by PEP 458 supports continuous delivery of distributions (because they are signed by online keys), but that model does not protect distributions in the event that PyPI is compromised. In the minimum security model, attackers who have compromised the signing keys stored on PyPI Infrastructure may sign for malicious distributions. The maximum security model, described in this PEP, retains the benefits of PEP 458 (e.g., immediate availability of distributions that are uploaded to PyPI), but additionally ensures that end-users are not at risk of installing forged software if PyPI is compromised.

One W3C Linked Data way to handle https://schema.org/SoftwareApplication ( https://codemeta.github.io/user-guide/ ) cryptographic signatures of a JSON-LD manifest with per-file and whole package hashes would be with e.g. W3C ld-signatures/ld-proofs and W3C DID (Decentralized Identifiers) or x.509 certs in a CT log.

Also predates netplan. NetworkManager (another redhat tool to makes things "simpler", like systemd) is mentioned in part 3.

I've used Debian based distributions for 20 years with nothing more than ifupdown, resolvconf, and occasionally dnsmasq, but no distro for old men and all that.

"no distro for old men". Hah, that's great. I had a brief thought of a Javier Bardem as SystemD.

I did the same for years, with the addition of wireless tools when I switched to using laptops. With Buster and the additional networking features of systemd (and iwd for wifi), I was able to replace the whole stack with just 2-3 things. Much simpler in design and easier to setup and use.

I wrote a small gist on the setup a couple years ago that is still mostly relevant. In case you're interested... https://gist.github.com/eikenb/1e9faeb8210e639e8caa69e9832db...

99% of my machines (everything apart from one laptop) has no wifi. Systemd is constantly

Now of course I can return to basics and learn everything again from scratch. The problem with that approach is distros like Ubuntu frequently drop in new features which last 5 years, and then they're gone as they're no longer sexy. This reduces the payoff of learning something new, may as well just continue to fight.

I probably won't use ubuntu on servers going forward as they broke the installer for 20.04 (they got rid of the nice simple debian-installer, which just needed a new kernel/initrd and a preseed tweak every 2 years) and replaced it with some live cd monstrosity. Managed to avoid netplan on 18.04. It's a pain, as it means spending mental effort to make changes to something that just works, and the whole reason I use linux is because it has "just worked" for 2 decades.

It seems all these recent changes are aimed at people who want their linux machine to operate just like an OSX laptop.

You can use autoinstall instead of d-i - I vastly prefer it over the arcane d-i configurations. d-i is one of those things which you get working and never touch again in case you break it.

https://ubuntu.com/server/docs/install/autoinstall

I've used d-i with ubuntu since 6.10 and it just works.

Never managed to find a base pxe boot for autoinstall, but it's just one of the many straws

As I say, just one of the many reasons I'm trying to leave the tech industry, just like the unix greybeards were annoyed by young upstart linux people in the late 90s, these things are cyclical.

Without wifi it isn't very compelling vs. the good old /etc/network/interfaces setup, it's hard to get simpler than that. I still use that on my servers as well.

I use Debian myself and know that they don't change tech lightly and will keep supporting Systemd for the foreseeable future. So that part isn't a concern for me. Can't help with the Ubuntu installer, I only ever used it for cloud deployments with cloudinit.

Systemd is a mixed bag but overall I've been happy with it. I have been using it since Jesse and have never had any issues with it and definitely prefer it over sysv-init (which I always hated, every other init system is an improvement over it IMO... so low bar I guess).

You can get a "legacy" installer for Focal server:

http://cdimage.ubuntu.com/ubuntu-legacy-server/releases/20.0...

I'm using it myself for $DAYJOB, although, as you say, my Bionic preseed needed a _lot_ of adaptation.

I'm looking forward for "old men" to retire.

Stuff like systemd (and systemd-resolved) are wonderful innovation and that m god they're the de-facto standard on most desktops.

And by the way, the joy of systemd-resolved is in complex stuff that become surprisingly easy.

Example: split-horizon DNS. You're connected to a VPN network but you're only routing some subnets through that, not everything. Thus, you have to have DNS that query some DNS server inside the VPN for a set of domains, but keep using the usual DNS servers for everything else.

It's surprisingly easy with systemd-resolved, but I'm not aware of any feasibility without a DNS server running on the local machine (albeit in resolver mode). Which usually would be dnsmasq, but at that point you might as well run systemd-resolved.

> Example: split-horizon DNS. You're connected to a VPN network but you're only routing some subnets through that, not everything. Thus, you have to have DNS that query some DNS server inside the VPN for a set of domains, but keep using the usual DNS servers for everything else.

Why. I want to route 10.0.0.0/8 via a VPN, I don't know if "flargle.bazonga.com" resolves to 10.0.0.1 or 99.99.99.99 until I do the lookup.

You might know beforehand.

Your employer might have the whole intranet.employer.tld within the VPN. And everything within that network resolves to some 10.8.0.0/8 address.

I might, but that's not at all how my networks work.

Indeed even the corporate business network (which I rarely ever touch) doesn't do that any more - some internal looking DNS resolve to public IPs, some external looking DNS resolve to rfc1918s, some are overwritten if you're on the network or not, and vpns are now based around zscaler which doesn't even allow access via IP address, not entirely sure how it works but DNS resolves everything to 100.64.x.x or similar (cg nat I believe) - not sure how it routes web traffic to the right server, but as I say I very rarely use it (I don't think I've used it since covid started)

If I were setting up a corporate laptop, I would likely set all dns, and possibly all traffic, routing via the vpn for security/compliance/monitoring purposes. What benefit does resolvd give me?

If bazonga.com is domain only accessible through the VPN then you most likely always want to look up flargle.bazonga.com through the VPN name servers. On the internet you'll get NXDOMAIN.

However, this could be solved by setting the name servers always to the VPN name servers and make them recursive. But then, what if those name servers don't have internet access?

"I'm looking forward for "old men" to retire."

Well I'm a shade over 50.

I use Arch on my laptop. I love the flexibility of NetworkManager and I use a variety of DNS sources including a local unbound and a /etc/hosts file. I have a wired ethernet profile called RFC1918 with a lot of addresses on it that covers pretty much all router, firewall and switch defaults. It crucially has the check box for "Use only for connections..." in the routes section. That means that the default gateway stays on the wifi and/or VPN. I have several other profiles to deal with various combinations of requirements.

Recently I modelled a data centre on my home network to get a Fortigate cluster up and running as though it was in place. It's WAN was behind a VLAN on my home pfSense, so it had it's "real" external IPs running fine and then I put VMs in at various places to test out routing and policy before deployment.

DNS is simply (lol) a name to address mechanism and there is no need to get too het up about it. I simply make whatever view of the world I need to exist at the time I'm doing a job. I do this for IPv4 and 6.

It can get a bit challenging WFH. I have a site to site IPSEC VPN to the office and also a OpenVPN dial up VPN connection available. Customer sites are generally setup to allow access from certain IPs and I have to make sure I appear to be the right address. Even more fun is had with the quite large collection of sites with a subnet range of 192.168.{0|1}.0/24. Those I generally wholesale NAT and then for the really dumb devices at the other end or those with a broken routing table, NAT outbound on LAN.

"My eyes are bent my back is grey ..."

Bloody kids 8)

I use networkmanager on my laptop. 99%+ of my linux machines are not laptops so I don't need network manager on them.

I use dnsmasq on two servers, which has a few local dns entries in its host (either to block lookups for some sites like dailymail.co.uk, or to have an entry for "nas" pointing to 192.168.0.51) and upstreams the rest of the queries. I don't need a forwarder on the other machines on the network, they are told by dhcp to use those two servers when they are on my network, when they're on another network dhcp tells them to use something else.

I run syslog on a few servers listening to an anycast device, and syslog fires text to the local machine and that anycast IP.

Had a problem last night with a systemd-resolvd server, I'd added an entry to DNS, and it wasn't working. Of course that's because the entry hadn't migrated from the main dns server to the backup one (the cronjob runs every 20 minutes as it has done for over a decade), and systemd decided to use the backup server (which is very rarely used). That costs me time.

If I want fancy DNS resolving paths, I can choose to install a fancy forwarder.

(For WFH I do have a router I can plug my laptop and other devices like a pi into and everything is forwarded through work, including routes to 0.0.0.0/0. No natting involved. When I bring my VPN up on the road, everything is routed via the VPN (other than the vpn end point obviously). I wouldn't bridge my network normally, why would I bridge it with a VPN?)

Author here. I can’t believe I didn’t cover that as I spent ages fighting with it before writing this.

Writing those posts was incredibly cathartic, but I still left stuff out because I couldn’t get to the bottom of it. I do recall that configuring systemd resolved and finding useful docs for it was a nightmare.

I recently had to circumvent the systemd resolvd auto config for dns it would randomly fail.

Quite normal - https://askubuntu.com/questions/907246/how-to-disable-system... for example

I actually went on to read this whole series a few months ago in search of what was wrong with some networking configuration I had at home. To no fault of the author, I felt like I trully learned a ton about DNS but couldn't apply that knowledge to solve my problem since ubuntu has added more stuff on top of these.

I feel like the DNS situation is way too complicated, I really don't even know where the tunnel ends anymore. I resulted to using the GUI and hope it handles everything, which I hate because I want to understand what's happening under the hood. I felt powerless

In Kubernetes you can also add NodeLocal DNSCache [1] as a DaemonSet that injects iptable rules to cache DNS locally to offload the in-cluster coredns.

[1] https://kubernetes.io/docs/tasks/administer-cluster/nodeloca...

many of you complained that there is no lightweight way on Linux to have different resolvers for different domains/zones (eg. one default and one for your VPN or whatever).

see https://github.com/azukiapp/libnss-resolver - it's an NSS lib just like libnss_dns or libnss_files, which looks up resolv.conf-style files according to the queried domain suffix.

it's not a deamon.

i integrated it with dhcp client and ifup/down and use it happily for years.

disclaimer: i contributed to libnss_resolver.

I always forget about strace. I'm gonna use that trick

though the systemd-critique-people/hater have some vlaid points, i nowadays tend to ignore them.

most of them do not contribute in building better solutions, which help more people then themselves. just blabbering on and requiring stuff which only works for THEIR usecase. this is really annoying at times.

Reply on news.ycombinator.com