Musl 1.2.4 adds TCP DNS fallback
openwall.com
openwall.com
I will admit though, that Debian-slim still has some non-essential stuff that usually isn’t needed at runtime, a shell is still neat for debugging or local development. This trade off could be considered a security risk, but it’s rather simple to restrict other stuff at runtime (such as run as non-privileged, non-root user with all capabilities dropped and a read-only file system except for /tmp).
It’s a balancing act between ease-of-use and security. I don’t think I’d get popular with the developers by forcing them to use “FROM scratch” and let them figure out exactly what their application needs at runtime and what stuff to copy over from a previous build stage.
Most of the reason I switched from Ubuntu -> Arch is that working with alpine allowed me to realize that installing packages and their dependencies don't have to take so long.
Essentially, dpkg is hilariously slow due to its approach in issuing fsync calls for every file - eatmydata wrapper injects a shared library which turns that into a no-op.
I haven't timed it rigorously, but I experienced up to 10x speedup in container builds by just wrapping dpkg with eatmydata.
I’m not the OP, but in an example I just tried it took build-essential from 48s to 39s (timed on the third step, without and then with the call to eatmydata):
from debian:sid
run apt update && env DEBIAN_FRONTEND=noninteractive apt-get install --yes eatmydata
run apt update && eatmydata env DEBIAN_FRONTEND=noninteractive apt-get install --yes build-essential
I ran it three more times. Plain install was 50s 48s 47s; eatmydata was 46s 46s 46s.
Installing build-essential probably does enough non-dpkg IO to account for the difference (install scripts etc, for example when libc changes and the system scans for running daemons) but it isn’t much and the delta may well be statistically insignificant.
- Docker image: `debian:stable-slim`
- `docker system prune -a` ran before each test.
- Packages installed: `build-essential ca-certificates libssl-dev software-properties-common wget curl git`
Results:
Default Debian settings: 48.32 seconds
force-unsafe-io enabled: 55.86 seconds
eatMyData enabled: 35.04 seconds
I haven't really built debian-based containers in last two years.
Also, Ubuntu 22.04 is only 28.17MB compressed right now so it looks equiv to debian-slim. There are also these new image lines, I can't recall the funky name for them, that are even smaller.
You might be thinking of the chiselled images. An interesting idea but very much incomplete[1].
Especially when you’re also developing within the container as well, having that be unified is absolutely worth the convenience, and honestly security and reliability as well. I realize that a container with less installed on it is inherently more secure, but if the only people who are familiar with the system are a small infrastructure/platform/ops type of team, things are more likely to get missed.
What would you suggest I restrict at runtime and can you point me to a tutorial or an article where I can go have a deeper read on how to do it?
In your Dockerfile, add a non-root user with UID and GID 1000, then in the end of your Dockerfile, right before a CMD or ENTRYPOINT you change to that user.
In your kubernetes yaml manifest, you can now set runAsNonRoot to true and runAsUser & runAsGroup to 1000.
Then there’s the privileged and allowPrivilegeEscalation fields, these can nearly always be set to false unless you actually need the extra privileges (such as using a GPU) on a shared node.
Then there’s seccomp profiles and the system capabilities. If you can run your container as the non-root user you’ve created, and you don’t need the extra privileges then these can safely also be set to the most restricted. Non-privileged non-root is the same as all capabilities are dropped.
The tricky one is the readOnlyRootfilesystem field. This includes /tmp, which is considered a global writeable directory, so the workaround is to make a in-memory volume and mount it at /tmp to make it writable. Likewise, your $HOME/.cache and $HOME/.local directories (for the user you created in your Dockerfile) are usually used by third party packages, so creating mounts here can be useful as well (if for some reason you can’t point it to /tmp instead).
[1] https://kubernetes.io/docs/tasks/configure-pod-container/sec...
I'm not certain, but would it be more secure to create a non-root user in the Dockerfile, but _don't_ use UID 1000. Use volume mounts to grant host disk access instead.
In our case, we have our own base container images that include this user. The base images are either just plain copies of official ones (such as dotnet or curl) or also includes stuff that almost everyone needs (such as MSSQL ODBC drivers, libxml2, curl etc) in the case of python/R/node.
https://cheatsheetseries.owasp.org/cheatsheets/Docker_Securi...
Our images come without a shell or pacakge manager by default, but there are -dev variants that include these.
https://github.com/wolfi-dev/ https://github.com/chainguard-images/images
https://www.chainguard.dev/unchained/scaling-chainguard-imag...
Pinning a language version (say python 3.11) isn’t an optional thing its a best practice, and the notion that its because of security seems intentionally misleading as the images should be refreshed in place on the tag along with signatures.
To be clear, nothing has changed with Wolfi. Wolfi is an open source community project and everything is still available there: https://github.com/wolfi-dev/.
We have made changes to Chainguard Images - our commercial product built on top of Wolfi - which mean you can no longer pull images by tag (other than latest). Chainguard images are rebuilt everyday and have a not inconsiderable maintenance cost (and the money we make here directly helps us support Wolfi).
The easiest way to avoid this is to build the images yourself. You can rebuild identical images to ours using apko and the source files in the images repo e.g: https://github.com/chainguard-images/images/blob/main/images... (note you can replace package names with versioned versions). You can also just use a Dockerfile with the wolfi-base image to "apk add" packages. Full details are here: https://www.chainguard.dev/unchained/a-guide-on-how-to-use-c...
I agree that pinning is a best practice. The above blog explains that you can still do it using a digest, but I accept this isn't the simplest solution.
If I can help any more, please feel free to get in touch - you can find me most places including twitter https://twitter.com/adrianmouat
else, frankly you claimed wolfi is oss, get customers to use your registry, and then bait and switched your early adopters.
aka, major version upgrades at random, have fun!
The built Chainguard images are all on the cgr.dev registry.
Wolfi is completely OSS.
The policies regarding Chainguard Images have changed over time, so if there are docs that don't properly reflect this, please let me know and I'll get them updated.
Given that it's a different layer, the your container runtime isn't going to redownload the layer anyway, right?
That means duplication of musl in lots of containers, but when they are all less than 10MB its less of an issue. Linking against gnu libraries might get the same executable down to less than 2MB but you'll add that back and more for even the tiniest gnu nase images.
Have you benchmarked musl vs glibc in any way? Data I've seen is all over the place and in curious about your experience.
Edit: realized I didn't answer your question, I ran flamegraph both ways and it was completely dominated by tight vectorized loops and a couple sadly necessary memory copies.
This has been an issue plaguing Alpine for years where the musl maintainer basically said the standard says may fallback, not must fallback. Let the rest of the internet change, we don't feel this is important. We're standards complainant.
It gained traction for the change with the latest RFC. for dns last year which made TCP fallback mandatory [0]. The cloak of standards compliant could no longer be used.
While I wouldn't call myself a fan of Postel's Law (I think "being liberal with what you accept" can allow others to get away with sloppy implementations over long time periods, diluting the usefulness of standards and specifications), I think at some point you have to recognize the reality of how things are implemented in the real world, and that refusing to conform to the de-facto standard hurts your users.
The fact that the maintainer only caved because the TCP fallback behavior is finally being made mandatory, and not because he's (very belatedly) recognizing he's harming his users with his stubbornness, also speaks volumes... and not in a good way.
> the maintainer only caved
> he's harming his users with his stubbornness
A gentle reminder that you're not entitled to other people's free labor. It's one thing to disagree with technical decisions made by the project, but the shaming and name calling is a bit too much.
And while I’d agree in principle that nobody is entitled to anyone else’s labor for free, I’d also add that nobody is entitled to be shielded from criticism of work they have chosen to publish, or to have that criticism be watered down to a sanitized, milquetoast version of the writer’s intended message.
Resorting to personal attacks because someone refused to implement and maintain your pet feature for free fits the description of "entitlement" in my book.
You certainly demonstrated the fallacy. You took the most absurd interpretation of my words that is possible in a literal sense which happens to be easier for you to mock.
Describing someone's behavior as entitled doesn't amount to making a false claim. That's the most obvious interpretation of what I actually said. But you took it to mean that any claim of entitlement can't be false.
> Your imagining someone to be entitled does not make them so, and your accusation belittles their actual, valid reasoning.
There's no imagination or accusation involved here when the very comment I'm referring to is right here in this very thread. It's a statement of fact.
The parts I quoted are ad hominem attacks that accuses someone of neglecting an obligation that objectively doesn't exist. That's the dictionary definition of entitlement.
> It actually shows you to be the one "entitled" to your projected redeclaration of their identity and rationale
Like misrepresenting my words or calling me "narcissistic?"
> potentially shared value-based opinion about the facts being criticised
The parts I quoted isn't a fact-based criticism. It's an ad hominem attack.
> irrellevant as calling out someone for being "entitled to breathe air" if you catch them criticizing the air's quality
Everyone is entitled access to clean air. People have an obligation to keep the air clean.
No one is entitled access to free labor, including having your pet features implemented and maintained in a software project.
But overall, gosh. How many fallacies, misrepresentations, and personal insults can you pack in such a short comment?
> intentionally-faulty software
> the maintainer only caved
> he's harming his users with his stubbornness
is strong language and not productive. If someone at my company sincerely criticized my work in this way I'd find it unacceptable and managers would get involved. There's criticism and there's polemics, and this is the latter.
I would agree there is a line, but disagree that the original poster stepped over it. Disagreement, even vigorous disagreement is important.
Calling out bad technical decisions is important.
Calling out oversensitive people who want to censor online discourse to their personal tastes is also important.
Fair, but I'd hold HN to a higher standard. We're peers and we choose to be here. We should treat each other with kindness and respect, not contempt. I read a fair amount of contempt in OP's characterization, and I hold that separate from respectful criticism.
I looked up a thread about this from 2020 [0] and it actually seems well reasoned, calm, and respectful. No one's slinging around words like "faulty", "broken", or "harm" because those tend to insult people and derail technical discussions.
And FWIW, that's why I would get managers involved if this happened in the workplace. Calling people's work "broken by design" or "harmful" is how to start an argument, not how to solve a problem. If someone's acting this way, there's no way things are going well.
Furthermore, ad hominem attacks have nothing to do with "calling out bad technical decisions."
BTW go take my words verbatim and use it against me if my actions are genuinely hypocritical. But to do that for my actions you made up is abusive and makes you look bad.
If that maintainer abandoned the project, it would either be picked up or abandoned altogether if the project were worthless.
It's a borderline case because it's almost worthless due to the aforementioned issues.
See RFC 1925, The twelve networking truths:
(1) It has to work
It must be nice living in your. theoretical world. Here in the real world, I like libraries to actually help me do what I want to do rather than be needlessly annoying to take a stand no one actually cares about. But that’s me. Clearly it serves Musl right given the massive amount of use it’s seeing. Hmm, wait a minute…
I suspect that the primary use cases for that would be applying half-baked workarounds and refusing to work with an "unsupported" libc, much like HTTP User-Agent.
I’m only half joking. Features detection in Musl is a generally pain. I don’t blame developers for sometimes taking the path of least resistance and not trying to support it.
http://git.musl-libc.org/cgit/musl/commit/src/network/res_ms...
AOL was the specific case we ran into that at the time (ca. 2000) returned either MX or A records (can't remember which caused the problem) that required more than 512 bytes.
Qmail's issue was a hardcoded buffer size so this patch to musl wouldn't have helped.
However as I noted in my other reply, musl also implements the res_* functions, like res_send() etc., and those can be used to address both.
I've never really understood the underlying mentality that causes maintainers to bend over backwards to _not_ provide popular functionality like this.
Reminds me a bit of the strlcpy/strlcat debacle with glibc.
What would they be doing with it that they couldn’t do better (faster/less bugs/risk) another way?
I think this one might be dumb too which is why I’m asking.
Once upon a time, the DNS spec required requests come from port 53. This was terribly insecure, so many sysadmins patched their DNS server to not do this-- most servers at the time didn't care (including the incumbent), and the few that did were easy to convince of the error of their ways with a simple demonstration.
Then one day, a DNS server became quite popular that didn't do this by default, and perhaps because the DNS spec was written by a company that sold consulting for their (buggy) DNS server, they used this exact argument: that it was wrong because the spec said it was wrong. They patched their server to reject these messages and encouraged users to block this traffic.
That was wrong of them, and eventually, due to enough pressure they eventually caved. That's because the reality of good engineering doesn't give two fucks what the spec says.
So what's your use-case?
Is this HTTP/S? SNMP? Something bespoke? Why doesn't the client already know where all the servers are? How far away are the servers from the client? Can you put code that lives in the client, or is this existing client software you're trying to trick into supporting client-side load balancing?
Also, and only slightly related: Do you have a different definition of "use-case" than me that involves being as vague as you possibly can?
Probably trying to prevent feature creep; musl's home page describes it as
> musl is lightweight, fast, simple, free, and strives to be correct in the sense of standards-conformance and safety
And every single feature they add makes those things harder.
That they only just implemented this is a joke, and Alpine/musl users are the punchline.
Or in other words they've always been standards-compliant, and when the standard changed they updated to match it.
"RFC" is Request For Comments. Some RFCs become standards. But there are no internet police, notwithstanding that the internet is a fascist state (run by corporations for the benefit of corporations). Understanding it as a fascist state is the rational way to view "enforcement" as well as the natterings over whether something is a standard and who is going to enforce it anyway.
https://github.com/NuGet/NuGetGallery/issues/9396
Our CDN provider ended up having a shedding mode in some hot areas that made the chain exceed the limit from time to time. Our multi CDN set up saved us so we could do geo specific failovers.
Asking about Alpine in a production environment was always good way finding who has container experiences of watching C-Beams glitter in the dark to those who only just read a "10 Docker Container Tricks; #8 will blow your mind!" blog post from 2017.
DNS is a fun rabbit hole for interviews, for sure.
My favorite one to see on a resume is NIS. If you are listing NIS and don't have horror stories or other things to say about NIS, that's a really good indicator of the value of your resume.
I intentionally list NIS on my resume because it is such a fun conversation topic to go on about how security models changed over time, all the ways NIS is terrible, but also how simple and useful it was.
For DNS, my favorite interview question goes like this,
How would you verify DNS is resolving from within a pod on Kubernetes?
After listening to the answer, add some constraints:
1. Common networking utilities like ping, nslookup, dig, etc are not available
2. Container user is unpriviledged
3. su/sudo do not work
This can lead to some elaborate k8s troubleshooting or the simple, and correct, answer of getent hosts.
I don't know Kubernetes, so it's very possible that Kubernetes enforces that name resolution is always DNS and only DNS (a "hosts: dns" line in /etc/nsswitch.conf, without any additional stuff like files which would mean resolution via /etc/hosts), in which case getent is indeed correct, but in the general case, there is no system-intrinsic tool that tests DNS only, and you have to use something like dig.
Lol.
This must be an IT thing because a developer would go, "ah, what is this shit? Hang on, I did this once back in 08', let me being up the documentation' and proceed to bootstrap a production-fix by EOD.
As an icebreaker, I pick out items from their resume that stand out and look like strengths. My goal is to play to those strengths and get them comfortable talking about their past battles.
1. Most people can do the job. Even programming isn't that bad, I've met very few complete incompetents.
2. Their personality and interest in the topic are better correlates for their success in the team and generally.
I'm still not going back until I can configure for TCP to be the default.
I dunno, I've been running containers in prod for a while now and I don't recall Alpine being a problem. Maybe it varies by your workload?
It's usually the case that everything works until it doesn't. When it's DNS that doesn't work, good luck debugging it unless you've got war stories to tell.
The result is images the same size as Alpine, or smaller, without the incompatibilities. I think Alpine is a dead end.
IMO Distroless or even scratch is nice for statically complied binaries or self contained deployments, but if there's a dependency on user space then it becomes complex.
Yes, some (I hope more rare now than when I wrote the FGA on it almost 20 years ago) firewalls cannot handle large DNS/UDP datagrams, but this code isn't for traffic that crosses firewalls. It's for traffic that goes only as far as the local resolving proxy, or at least goes somewhere "nearby" like 9.9.9.9. EDNS0 firewall worries are a lot less here.
And EDNS0, at least for the bit that enables >512 byte datagram sizes, is a lot less complex to implement than full DNS/TCP fallback. It's just proper buffer size handling and an extra resource record set.
To be clear, that's not "wrong", but just different maybe from what docker was expecting.
In any case glibc uses NSS, so what glibc does depends on the configuration. It may well just forward the request to systemd-resolved.
i’ve started using alpine on my laptop and ec2 because it’s not.
different strokes, different folks.
Musl 1.2.4 Released–Supports DNS Transport over TCP (openwall.com)