Some ways DNS can break | Hacker News Reader

Some ways DNS can break

jvns.ca

106 pointsvinnyglennon43 comments

This article ignores the most common problem I have with DNS these days which can roughly be construed as: What is actually handling resolution on my system?

We've come a long way from good ole fashioned libc/glibc acting as a good ole fashioned stub resolver on my UNIX system. Is systemd even looking at /etc/resolv.conf anymore? Who is overwriting /etc/resolv.conf? Is the information provided via DHCP being ignored? Is Firefox using DoH? Where is caching happening? How do I clear it?

All of these questions have completely different answers depending on OS and how they're configured. I work with OSX, Debian, Redhat and OpenBSD on the regular and they all have different answers for the above questions.

> This article ignores the most common problem I have with DNS these days which can roughly be construed as: What is actually handling resolution on my system?

This does my nut in too - even just narrowing the Q down to modern Linux Desktops with e.g. chrome/firefox, and I have absolutely no idea what the query path is. So many components have been added to the path over the last few years :(

> Who is overwriting /etc/resolv.conf?

    echo 'nameserver 127.0.0.1' > /etc/resolv.conf
    chattr +i /etc/resolv.conf

You can thank me later.

> A file with the 'i' attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file, most of the file's metadata can not be modified, and the file can not be opened in write mode. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.

So you essentially block any program trying to write it?

Yes. There’s software, like dhcp clients, that has claimed the right to automatically edit this without leaving a trace. It’s really hard to turn off, it’s easier to just ban this behavior categorically, if your use case is not the one in which this behavior is useful.

This. I find it inexcusable that the Debian udhcpc package still does not provide any configuration option to inhibit it from overwriting /etc/resolv.conf.

The only option is to replace the "execute this when DHCP reply is recieved" script, which is not a simple script at all. And then you're stuck three-way-merging your changes to that script with every security update.

yeah that breaks my fucking VPN which won't start unless it can overwrite resolv.conf.

So now I have a job that watch for that and does a bind mount to make everything work.

Get a better VPN, like wireguard.

In case of Linux with glibc, check /etc/nsswitch.conf. The line in there starting with "hosts:" has the configured order of host name resolution methods, where "files" means /etc/hosts, and "dns" means /etc/resolv.conf. Other entries, like "mdns" or "mymachines" are additional optional modules and you should consult their respective documentation, and you can always uninstall them if you don’t want them.

Full documentation: https://www.gnu.org/software/libc/manual/html_mono/libc.html...

Indeed. This was a good post a while back on just this issue:

https://tailscale.com/blog/sisyphean-dns-client-linux/

> dig ignores /etc/hosts

Yes, when debugging host name resolution for a specific host, don’t use dig; use “getent hosts <name>”. This will use the normal configured host name resolution path. (Personally I use "getent ahosts <name>" to get both IPv6 and IPv4 addresses.)

Use nslookup. The dig tool intentionally goes directly to DNS and bypasses all local overrides like NIS, /etc/hosts, etc….

I didn’t know about the dig behaviours but when I read about it I immediately thought that is a great way to narrow down the problem - compare dig results with something else that does respect local conf.

That is exactly how I always used it. If dig provided different answers than nslookup, then we knew where to start looking.

nslookup also goes directly to DNS, as far as I can tell. From what I remember, nslookup is what people used to use before “dig” became popular. So there is no good reason to ever use nslookup.

It depends on how your nslookup application is configured. They can be configured to go straight to DNS and not use any local host resolution, that’s true. And in that case, I consider it to be marginally useful — at least it generates less noise than dig. However, by default on most platforms, nslookup is specifically configured to use /etc/hosts (or the equivalent) or other local lookup facilities (like NIS) before using DNS. That’s the real reason you use nslookup, to see what those other local name resolution facilities might tell you that are not represented in the DNS.

> However, by default on most platforms, nslookup is specifically configured to use /etc/hosts (or the equivalent) or other local lookup facilities (like NIS) before using DNS.

I can find no reference to this, nor does nslookup do that on my system. Maybe on Windows, its “nslookup” (which is not the real Unix nslookup) command does this?

See https://en.wikipedia.org/wiki/Nslookup

In particular, note the following quote: “Unlike dig, nslookup does not use the operating system's local Domain Name System resolver library to perform its queries, and thus may behave differently. Additionally, vendor-provided versions may include output of other sources of name information, such as host files, and Network Information Service. Some behaviors of nslookup may be modified by the contents of resolv.conf.[2]”

Bonus error: you can't have DNS packets >64k. Even over TCP. (Been there)

Someone has configured a delegation loop. (Yep, been there more than once)

Records are currently unavailable, but someone has busted the cache. (Enjoy)

All services wait for the same DNS record to be fetched and will all ask the same service at exactly the time when DNS is done. (Again, clear yes, happens. That's why you should use prefetch)

No one knows how to handle DNS SRV priority correctly (please correct me if I am wrong).

The problem with DNS SRV priority is that it rarely provides the behaviour you actually want.

I remember when you could list entire cctlds, how did that work then? They weren’t as big as they are today but definitely bigger than 65k.

It sounds like you're thinking of zone transfers (the special AXFR query type). In that case the server's response may consist of more than one DNS message sent consecutively over a single TCP connection [1], so it's possible to transfer large zones. E.g. out of curiosity I just asked for the root zone with

  dig . axfr @iad.xfr.dns.icann.org

and I received 24,448 records transferred over 83 DNS messages, according to the stats dig prints at the end.

[1] https://datatracker.ietf.org/doc/html/rfc5936#section-2.2

>Bonus error: you can't have DNS packets >64k. Even over TCP. (Been there)

I am curious how did you get there given the length field in DNS records is only 16 bits?

I think the point was that they wanted to return more data than 64kB in the response but couldn't.

SRV is deprecated. Don’t use it at all.

When did SRV become deprecated?

The standard[1] doesn’t show it was obsoleted by anything else.

1: https://datatracker.ietf.org/doc/html/rfc2782

I think it was one of the BCPs that recommended that people not use SRV records. I’ll have to check.

"Why do people find DNS so difficult? It’s just cache invalidation and naming things."

-- Jeff Waugh

https://twitter.com/jdub/status/739110670562557952?lang=en

I read his name as "Jeff Laugh". I assume that's what he was doing when he wrote that tweet.

On the `ndots` problem, it's best to get in the habit of using fully qualified names when you know them. A fully qualified DNS name ends in a period.

Continuing the example in the article, use `google.com.` and there will be just one query.

You can kinda fix this by lowering the ndots value. But probably the better way to fix this is to not include a 'search' directive at all. Don't specify anything for 'search' or 'ndots'. And then do as you say, use FQDNs everywhere.

That's not always possible & wise, e.g. the example given is the Kubernetes search/ndots values. Changing this would break the expected "API" between the applications in Kube and the Kube cluster, which is bound to cause some pain and suffering down the line :)

Using a FQDN (as in, a DNS name with the trailing dot) is the only reliable way to avoid this without breaking other things!

This doesn't work reliably, because TLS gets implemented incorrectly and the trailing dot is not stripped from the SNI host (TLS not HTTP) header.

Curl has gone back and forth; dunno whether that is curl itself or the resolver libs. Firefox used to be broken, and may still be.

Apache used to almost blow chunks when it got one of these, and I filed a bug (https://bz.apache.org/bugzilla/show_bug.cgi?id=58007); the chosen fix was to serve some "random" cert. Just tried it with an older version of FFX which I know is borked though and it did the right thing, so presumably something further has occurred since 2017.

One recent frustration I ran into with Node's DNS is that it doesn't seem to resolve whatever.localhost to 127.0.0.1, unlike my browser.

Not entirely sure which is doing the right thing by the spec here, but I much prefer the browser behavior, because I would really like to not have to add a hosts entry for everything I reverse proxy from Caddy for doing local HTTPS/HTTP2 development for production parity.

Oh, and I also found out that Node also isn't able to use the system-wide certs store (which local HTTPS in Caddy adds to) [1], so in the end even adding a hosts entry didn't work, and I had to revert to plaintext HTTP1. Deno is looking more attractive every day [2]. Too bad I can't use it for building VSCode extensions.

Really curious to hear how other folks are doing local HTTPS/HTTP2 setups in node these days.

[1] https://github.com/nodejs/node/issues/3159

[2] https://github.com/denoland/deno/pull/11491

Node (community?) does a lot of weird things when it comes to localhost. No idea if this is still true, but a lot of libraries/servers bound to 127.0.0.1 instead of localhost. Watching it explode in an ipv6 environment was cute. Then there was Docker and that didn’t work because the socket wasn’t bound to a public socket. I don’t know what 0.0.0.0 wasn’t the default in these libraries. Maybe it was a hint it wasn’t to be used in production…

> resolve whatever.localhost to 127.0.0.1

I think that's the browsers making names up. No system resolvers I've seen respond to an x.localhost lookup.

I generally have a wildcard setup on an internal domain for generic local names (*.local.me)

> Node also isn't able to use the system-wide certs store

Node can use the `NODE_EXTRA_CA_CERTS` environment variable to add a file of one or more PEM trusted certificates at launch.

https://nodejs.org/dist/latest-v16.x/docs/api/cli.html#node_...

> I think that's the browsers making names up. No system resolvers I've seen respond to an x.localhost lookup.

Fair enough, I guess I just wish node would behave more like a browser in this case then, as it'd have saved me a lot of headache.

> I generally have a wildcard setup on an internal domain for generic local names (*.local.me)

Curious how you set that up? Hosts files don't allow wildcards afaik, and that's what I've been struggling with.

> Node can use the `NODE_EXTRA_CA_CERTS` environment variable to add a file of one or more PEM trusted certificates at launch.

I actually tried this option briefly but couldn't get it to work, at least not with the root cert that Caddy generated. I would much prefer a `NODE_USE_SYSTEM_CERTS` option that doesn't require pointing to any specific paths though, as there's less room for misconfiguration, which I can't rule out in my case.

> Curious how you set that up?

Via a DNS server we run, that then answers for local things or forwards requests on to the real world.

To do that locally, set up a DNS forwarder like dnsmasq [0] or unbound [1] (or coredns/bind/etc) to respond for the domain you want to wildcard to localhost. Point your OS DNS settings to 127.0.0.1, then configure the forwarder with your regular external DNS servers.

The equivalent to `NODE_USE_SYSTEM_CERTS` is a build time option unfortunately. I know the rhel/debian packages do integrate their builds with the system CA's but those packages tend to be ancient. Maybe try the nodesource [2] packaged node.js? The homebrew node formulae [3] looks like it integrates with the brew openssl CA's.

[0] https://stackoverflow.com/a/22551303

[1] https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound...

[2] https://github.com/nodesource/distributions

[3] https://github.com/Homebrew/homebrew-core/blob/c2579d98415bd...

There is the suggestion made in the article that you can get any random IDN, and to the best of my knowledge that is not true. There's something about collating characters (there is no collation order for poop, hearts or astrological signs) only, and it has to be in only one character set (no mixed languages).

That said, you are free to create "<poop>.example.com", presuming you own example.com. Indeed, "`rm -rf *`.example.com" is a valid FQDN according to the DNS (protocol) as opposed to The DNS (implementation).

Like last time I'll post a simple snippet of Java with dns4j that solves this weeks question:

>problem: Java caching DNS records forever

  SimpleResolver resolver = new SimpleResolver("8.8.8.8");
  Name name = new Name(host + ".");
  Record a = Record.newRecord(name, Type.A, DClass.IN);
  Message query = Message.newQuery(a);
  Message response = resolver.send(query);
  Record[] answers = response.getSectionArray(Section.ANSWER);

problem: systemd-resolved fails randomly: https://github.com/systemd/systemd/issues/19118

- this happened to me on two arch linux installations for different domains - I was unable to debug this - would love to somehow get this fixed but I switched to dnsmasq that works fine for me.

Also, throw in all misconfigurations related to DNSSEC and you are in hell.

Reply on news.ycombinator.com