DNS resolution issue in Alpine Linux (2021) | Hacker News Reader

DNS resolution issue in Alpine Linux (2021)

christoph.luppri.ch

90 pointsnelsonmarcos58 comments

We're a somewhat popular hosting provider that runs Docker containers (as VMs) for our customers and does private networking over IPv6, which expands the size of our DNS requests, and we run into this all the time with Alpine. It's kind of baffling.

TCP DNS is not hard. It's part of the spec. Normally, that argument doesn't mean much to me --- lots of things are parts of specs that I think are silly and not worth doing --- but TCP DNS seems like a basic necessity for DNS to work at all.

What's holding this up? TCP DNS is just UDP DNS, but over a TCP connection, with the packet length sent before the packet itself. It's the simplest thing you could possibly come up with to make TCP DNS work. It's been there since the 1980s. They should add it.

Possibly ask on #musl or submit a patch?

They know about it! I could write the patch, but they're not going to accept it; there's something weird going on about this.

There isn't anything weird, I went on #musl asked the question, the functionality is desired but it should be worked with the community to ensure correctness.

The functionality was explicitly not added by the developer.

> My choice not to do TCP in musl's stub resolver was based on an interpretation that truncated results are not just acceptable but better ux - not only do you save major round-trip delays to DNS but you also get a reasonable upper bound on # of addrs in result.

https://twitter.com/RichFelker/status/994629795551031296

Maybe he is willing to consider it now, I don't know.

It's not a bug, it's a feature! You're welcome. (tm)

Yes, there is. I'm not just making this up. I've gotten takes from "musl people", including Rich Felker, that this was a deliberate choice, and that not only is musl's behavior of pretending that truncated UDP responses are valid acceptable, but that it's somehow better (maybe for performance?) --- that you can always serve "getaddrinfo" off of just the truncated UDP result. Which is, of course, false.

There is a weird thing going on; maybe you can relay this back to the #musl channel you're talking on, and they can keep discussing.

> There is a weird thing going on

It's not weird they're just incompetent / a bozo.

I mean sure, musl should add TCP DNS. But real world networking says if your DNS response is over the limit for basic UDP DNS, a meaningful proportion of clients won't get that response. Which I guess is okish if it's a TXT for mail servers, cause mail servers probably have a reasonable DNS setup; but if you wanted to return a lot of A records or AAAA records (or A records that a provider DNS64s into AAAA records), you need to be careful about how many you return, before the response gets too large and doesn't arrive.

This is something networks and DNS client libraries should fix, but we live in a world where PathMTU only mostly works, so realism gets your service working.

It's non-trivial to determine if your response will fit into the UDP packet size, other than erring heavily on the side of caution.

Part of the request query is included in the response, IPv6 AAAA records are considerably larger than IPv4 A records, DNSSEC adds size, RFC 1035 message compression comes into play, EDNS0 comes into play, etc.

Which is all to say that you're right that for maximum compatibility you should endeavor to fit all responses into a UDP packet, but most places lack the sophistication to ensure that is always the case.

Supporting TCP DNS on the client side should not be considered optional if you want to successfully resolve records that you don't control.

That's acceptable in the real world because virtually nobody uses IPv6, and people who run into IPv6 problems will just avoid musl.

>virtually nobody uses IPv6

Some very high percentage of mobile devices, when using mobile data, are on IPv6 and preferentially connect using IPv6. I think 80% of Meta traffic is IPv6.

Lots and lots and lots of people use IPv6.

I have to say though the BOFHesque attitude shown in this thread by a number of people has convinced me never to touch Alpine linux. I can already see exactly what sort of attitude towards users and input the developers encourage and that sort of ivory tower shit has absolutely zero appeal.

This is literally the opposite of true. The OS-supplied resolver supports TCP DNS on all of the most popular desktop and mobile OSes and has for ages. I'd estimate a minimum of 90% of users have setups that support it, and I'm only hedging the 10% because I'm not sure how prevalent horribly misconfigured corporate firewalls are.

By not supporting TCP DNS you are the odd one out.

The OS supplied resolver may support it, but many networks block TCP DNS.

Many networks also block AAAA DNS, because middleboxes are stupid. That doesn't mean it's OK for libc not to implement AAAA.

You should be running a local resolver. I get where you're coming from but if you "keep running into this" you should probably stop running into this?

How running a resolver will fix `getaddrinfo` not using TCP ?

This is why I like glibc’s approach of making the actual resolution someone else’s problem and pluggable.

RFC9210: DNS Transport over TCP - Operational Requirements

https://datatracker.ietf.org/doc/rfc9210/

   This document updates RFCs 1123 and 1536.  This document requires the
   operational practice of permitting DNS messages to be carried over
   TCP on the Internet as a Best Current Practice.

The moment I saw Alpine Linux in the title, my first guess was "I bet this is something to do with musl libc". Briefly looking through the blog, it looks like my gut feeling was correct.

A while ago I evaluated Alpine Linux. I wanted to like it, I really did, it ticked so many boxes.

But time and time again, I kept on running into issues with their adoption of musl libc.

The last straw for me was when I discovered packages in their package repo (some of which were well-known names) that were compiled against musl when the upstream developers quite clearly wrote in their docs that "if you compile X against anything other than glibc, you're on your own". For me, the fact that Alpine ignored this and compiled against musl anyway, was a big red flag. (And yes I raised some of these as bug reports, but the cases got closed and nothing done about it).

"you're on your own" is like, the expected thing when you are doing something different.

Still, Alpine is on top of everyone else when it comes to Docker Images sizes. Thats why it will stick.

> "you're on your own" is like, the expected thing when you are doing something different.

Indeed. And that's fine. As long as you're willing to support that difference.

But the "Alpine compiling XYZ against musl" thing is/was just being done blindly by Alpine (i.e. load X into auto-build and let it rip). Sure it compiled without errors. But it never ran properly.

I've rarely had issues and if I do there is a process to report them to get them fixed. Not yet mad and act like the entire distro is worthless because I can't be bothered with reporting an issue for something that is already free. I've had less issues with Alpine than I've had with software on Windows.

If you care about image size you need to switch to a distroless image base. It's incredibly easy nowadays with multi-stage builds. IMHO people use alpine not because they seek it out but because they're reading old guides and advice that hasn't been updated with the times.

Blindly switching to alpine to save space is opening yourself up to exactly these sorts of musl-libc quirks and issues--you better have solid test coverage and a plan to make sure there aren't unexpected edge cases.

There are a lot of tools that simply can't run distroless and adding in all the requirements is often tedious with little reward.

What are some of the tools that can't run on distroless?

Anything that relies on system libraries. This works fine for most Go apps, but can have unexpected results if you don't compile the app correctly to use Go networking and what not.

I see. Regarding Go, how would forget to import the Go net and http packages for your app? I would think your compilation itself would fail to begin with. Or are you referring to something else?

In my time building Go apps in the past, you'd need to specify the netgo tag and others. Not sure if this is necessary anymore, but I can think of a few Go packages that require CGO and what not that will not work using a scratch image.

I would suggest to just use a small number of base images. The total size of all layerse is irrelevant in most cases if it's a small layer on top of something common for you = cached.

Are they still on to of everyone else in a meaningful way, though?

There was discussion yesterday [0] that pointed out that there are 29MB Debian [1] and Ubuntu [2] images.

[0] https://news.ycombinator.com/item?id=30633670

[1] https://hub.docker.com/layers/debian/library/debian/stable-s...

[2] https://ubuntu.com/blog/minimal-ubuntu-released

I run Alpine Linux from a 512 MB CF card, so yes.

What do you use this for?

On a Siemens Futro S-300. I use it as a Home Server. A 2-TB disk is mounted there, also a printer.

>The last straw for me was when I discovered packages in their package repo (some of which were well-known names) that were compiled against musl when the upstream developers quite clearly wrote in their docs that "if you compile X against anything other than libc, you're on your own".

(I assume you meant "glibc", not "libc".)

That's how every software works. The software developer cares about A, B, C distros at most, so other distros are on their own. The maintainer of distro D takes responsibility themselves to make the package work on D. The maintainer needs to understand the software well enough to be able to assert that it will work on their distro, patch it as necessary to make that happen, and maintain those patches in the light of bug reports from the distro users.

>(And yes I raised some of these as bug reports, but the cases got closed and nothing done about it).

Well, yeah. Unless you find something that is irrecoverably broken against musl such that it can only be fixed by compiling against glibc, your bug report is pointless.

> (I assume you meant "glibc", not "libc".)

Yes, corrected.

> The maintainer of distro D takes responsibility themselves to make the package work on D.

Yeah, but Alpine are not doing that, or at least not at the time I evaluated. I got tired of finding distro supplied packages that just didn't work as expected because things broke unexpectedly because of musl.

I think its important to recognise that these are community projects -- they don't owe you anything so don't misunderstand this "responsibility" you're citing.

If something is an issue that's important to you, these things get solved when people like yourself make it known constructively with test cases or bug reports.

There are better distros with package managers that allow multiple variations of libc to live together.

> [...] the standard was extended by two options:

> - Increasing the size of the UPD packet above 512 bytes via the Extension Mechanism for DNS (EDNS)

> - Switching the protocol from UDP to TCP

> Alpine Linux, or rather musl libc, doesn’t support either of those options.

It still seems weird to me that such details are decided by libc. My reflex idea when designing a system would be to put DNS functionality in a system service, while libraries would only query the service, without troubling themselves with system caches, TCP vs UDP etc. Then possibly the service could be even swapped for another with a compatible interface, but making different decisions, without perturbing the applications. It sounds like systemd-resolved is a move in that direction, but I still don't understand why putting all that in libc, essentially making all applications perform their own independent DNS work, was the original choice.

We've run into DNS issues with Alpine containers at two different places I've worked at. Completely different data centers and infra.

First time it took a lot of effort to pinpoint the problem.

Second time too, since it appeared because of a non-relevant code change (which lead to slighty more DNS requests).

In both cases, a simple switch to Debian slim saved the day.

Alpine is since banned from any env I'm working in :-)

DNS in Alpine is notoriously buggy but it can get months until you realise that. One easy and effective solution is to force dns resolution like so

dnsConfig: options: - name: ndots value: '1'

cc: https://support.cloudbees.com/hc/en-us/articles/360040999471...

There are also plenty of dormant issue, enough so that I won't be using Alpine ever again imo :'(

Few days ago, I spent quite a few hours trying to make `apk update` work for alpine on WSL2 on Windows. It didn't want to resolve dl-cdn.alpinelinux.org within alpine. Did resolve on host ubuntu.

1. WFH from VPN, firstly I had to lower mtu from 1500 to 1392 (My VPN specific issue) https://github.com/microsoft/WSL/issues/4698

2. Next, I had to run some powershell script that updates /etc/resolv.conf to use my VPN DNS (WSL specific stuff) https://github.com/microsoft/WSL/issues/1350

3. And I still don't know if apk works properly. Kind of works in Docker build, but I have a feeling something not quite right.

See this example. Why does it "hang"? Docker command not exiting

  docker run -it alpine:3.15 apk update
  fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz

Now, doing it within container itself, works:

  docker run -it alpine:3.15 sh
  / # apk update
  fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
  fetch https://dl- 
  cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
  v3.15.0-342-g4fee739486 [https://dl-cdn.alpinelinux.org/alpine/v3.15/main]
  v3.15.0-340-g4ed6115e99 [https://dl-cdn.alpinelinux.org/alpine/v3.15/community]
  OK: 15859 distinct packages available
  / # exit

Can someone shed some light?

Is DNS set in /etc/docker/daemon.json?

TLDR:

The solution to an intentionally broken resolver is to use a third party library.

Its only broken if you dont consider robustness a feature.

It's not just "robustness". Not supporting TCP DNS breaks DNS if your responses are "large", for values of "large" that include numbers that are in fact very small.

largest safe size to use is ~548 bytes - anything more than that and you need tcp

Or, as the article says, EDNS.

Its okay to ignore "modern" features if they create bad edge cases.

This feature is from 1986.

So are absolute domain names, but everyone is using relative domain specifications now, omitting the final dot.

HTTP Transfer-Encoding also got specified, and then collectively mis-implemented.

That its in the standard for decades doesnt mean it will be good when used.

I can use absolute domain names just fine in the software I use.

This doesn't make sense as an argument. Without TCP DNS, you're stuck with an untenably low limit for how much data can fit in a DNS response. Not having TCP DNS breaks DNS. It's not an aesthetic argument.

Things can be broken in more than one way not just solely when the thing broken is robustness (for some value of robust).

There are two problems here:

• musl should support EDNS and DNS over TCP/IP without issues

• People should be smart enough to use DNS services that don't have stupid edge cases

For the latter, if you use Google for resolving DNS, you get what you deserve. Run your own resolver if DNS resolution matters.

Ouch. This kind of issue seems like a real showstopper. It really makes me hesitate using Alpine.

Alpine has had DNS issues since the very beginning, but I am surprised to read that it still has some.

In my experience, the storage, bandwidth and time savings using Alpine Linux (even if they were much more significant than they are in practice) are not worth it given the issues you run into every once in a while. Just go with Ubuntu/Debian base images and you'll be much happier that you did in the long run.

Reply on news.ycombinator.com