Alpine Linux: Brilliant Linux Distro | Hacker News Reader

Alpine Linux: Brilliant Linux Distro

wiki.alpinelinux.org

123 pointslastdong92 comments

I really want to like Alpine, but we (Fly.io) have seen so many DNS issues with customer images that we’re now recommending Ubuntu or Debian slim. The extra ~50mb is a worthwhile trade off to avoid hard to debug musl-libc issues.

eg https://www.linkedin.com/pulse/musl-libc-alpines-greatest-we...

What about busybox? It still has a glibc version doesnt it?

Yep. I had an (immutable) JVM ECS service that worked fine for a year, then started failing DNS in just one of several AWS AZs and for just one of various host names--one with something like 24 A records, a few but not an outrageous number. Occasionally forking a process to run 'dig' on the same name made it work for a little while.

AWS support's only advice was "don't use Alpine"; annoyingly, switching the containers to a Debian base cured it, even though this would appear to make absolutely no sense with respect to it failing in just one AZ.

Failing in one AZ probably means they changed something with DNS servers for that region. We had a similar issue recently when we rolled out a new DNS server that returned longer SOA records which broke python's mysql driver in only some regions during the rollout. Debugging nightmare fuel.

Yes, my theory was also some DNS variance in the AZ, but AWS stolidly refused to supply any information to that effect. Adding 'dig +trace' to the equation only deepened the mystery.

Interesting!

Thank you for sharing. Did you notice that some language are more affected than others?

For example Go vs. Node.js?

Node, Ruby, and Python come to mind. Oddly enough the alpine versions of haproxy and nginx on Docker hub had issues too.

Any insight on Laravel/PHP? I've been seeing DNS resolution issues with Alpine as well, but they seem to mainly affect PHP services and not Go.

Also, isn't Python problematic with musl-libc [0]?

[0]: https://pythonspeed.com/articles/alpine-docker-python/

The person responsible for musl's DNS support thinks broken DNS is good UX and purposefully didn't add tcp upgrade support to it.

Note that some DNS resolvers do not provide truncated UDP results apparently so that might explain some of the weird DNS issues people see https://twitter.com/RichFelker/status/994629795551031296

> Keep in mind that today due to the great popularity of Docker, Alpine Linux is one of the most deployed operating systems currently in use, because within every other operating system that uses docker, the docker image it uses is almost always Alpine Linux.

Isn't this a little overstated? I'm willing to accept it has a majority, especially once you get into custom images, but e.g. docker's library images are usually debian based, with a -slim or -alpine variant available, and of developer provided images on my network, debian + derived images have a majority, while Alpine's statement leads you to expect some kind of 90% share.

Docker images used to be entirely Debian-based but have slowly become almost entirely Alpine based. A lot of Debian users have become Alpine users, and I've been mulling over if I should join as well (since I've used Debian for 20+ years but are friends with Alpine devs and users).

It was popular, but it has always been someone’s choice to start from someone else’s flat base.

So I surveyed docker images in my personal use:

Postgres (Docker provided - https://hub.docker.com/_/postgres): Debian (with alternates for alpine or other debian versions)

Jellyfin (Developer provided - https://github.com/jellyfin/jellyfin/blob/master/Dockerfile): They use an alpine build step but the final distributed image is debian

Calibre-Web (Linuxserver provided - https://github.com/linuxserver/docker-calibre-web/blob/maste...): Ubuntu

Graylog (Developer provided - https://github.com/Graylog2/graylog-docker/blob/4.2/docker/o...): Debian

Vaultwarden (Developer provided - https://github.com/dani-garcia/vaultwarden/blob/main/docker/...): Debian (with alpine alternate available)

I'm finding it hard to reconcile "Some of the software has secondary alpine images available" with "Almost all software uses alpine images"

For professional use, our company mandates all images used are built off a common base image, which is Ubuntu based (my previous employer was similar, but used a Red Hat based image).

We use alpine based images extensively where possible. Most popular language ecosystems and tools have an alpine option. Alpine as a base image is nice because it's nearly blank, but easy to customize unlike distroless. Recently one tool did drop Alpine, most based on it's reduced performance, and that was envoyproxy.

That used to be huge a few years ago.

But with both Ubuntu and Debian shipping much smaller images, I almost always find the hassle of using Alpine not worth it.

Same currently. Unless I'm just modifying a simple trusted image (which does happen often enough) I just start from Ubuntu during development and only attempt to move the image to Alpine if I'm really trying to improve resource usage down the line.

TIL there is a 29MB Minimal Ubuntu!

https://ubuntu.com/blog/minimal-ubuntu-released

Wow, wish there was a Debian version!. I miss JeOS!

There is a ~29MB Debian Docker image:

https://hub.docker.com/layers/debian/library/debian/stable-s...

Wow, I wish there was a Devuan version!

I've used Alpine pretty heavily but did not run into any hassle aside from "oh yeah forgot to install this package".

Could you go into more detail about the challenges one could encounter when using Alpine?

Alpine uses a different C standard library called musl compared to most well known distros that use glibc. You can run into issues if the application makes use of functions in glibc that musl has not implemented. Usually you run into them in applications that are written in C, such as in some Python modules that utilise C under the hood or in databases etc.

"I can't find $SOFTWARE in $PACKAGE_MANAGER now I have to compile it from source and my life sucks" has happened with me way more times with Alpine than Debian.

Why don't more people use distroless?

Based on debian so no musl headaches but even lighter than alpine

https://github.com/GoogleContainerTools/distroless

Because you cannot just jump in your container to debug. There are ephemeral containers for Kubernetes to add an image on top of a running container, for troubleshooting, but I haven't found something like that for Docker or Podman. You would need some kind of patchwork script where you untar an image and copy it inside the running container I guess.

WARNING: Blatant self promotion follows :)

I've recently switched to using Alpine on my laptop, and have written a bit about setting it up using ansible here:

https://www.brianlane.com/post/alpine-laptop/

And when you can't find the apps you need in the distribution there's always podman:

https://www.brianlane.com/post/prusa-slicer/

I ran Alpine on my laptop during the last half of college. It's a great little distro if you don't mind occasionally compiling things.

I've been on Gentoo since 2005. It's a great distro if don't mind compiling everything ;)

I got hazed by an upperclassmen with gentoo around that time and it's ruined me ever since. It's just not Linux if I can't edit a text file and rebuild the world.

Fun fact: Gentoo actually uses Alpine docker containers as a base to build their Docker containers.

Instead of using podman directly, and potentially messing with X11 permissions, you could use x11docker (despite the name, it podman support). It attempts to safely share X11 into a container.

Tried using x11docker in Alpine, didn't work for the Electron apps I tried with it. Tried for several days; finally got the app running in Docker without x11docker.

> safely

vs upstream saying

> Using x11docker as a sandbox is not intended to run obviously evil software.

Doesn't Alpine have a noticeable performance penalty due to being based on musl libc instead of glibc?

This seems to be born out in multiple people's micro benchmarks though how much of an issue it is for larger applications probably depends on the workload.

https://superuser.com/questions/1219609/why-is-the-alpine-do...

https://nickjanetakis.com/blog/benchmarking-debian-vs-alpine...

I've tried alpine's gcc before and it was about 50% slower to compile a bunch of packages. That said, if you compile llvm with MinSize/-Os on glibc distro's it runs 50% slower too, so it's unclear if slowdown is due to musl libc or rather that they optimize for size instead of performance.

Compiling is CPU-bound and doesn't really bottleneck on anything that libc is involved with other than the allocator, so unless musl's allocator is comically slower than glib's, chances are it's just -Os.

I believe musl’s allocator is known for being pretty poor.

musl-1.2.1.tar.gz (sig) - August 4, 2020

    This release features the new "mallocng" malloc implementation, replacing musl's original dlmalloc-like allocator that suffered from fundamental design problems.

https://inbox.vuxu.org/musl/20200518194204.GG21576@brightrai... Pretty good writeup from the author

https://musl.libc.org/releases.html Rest of the detail in the actual changelog is worth reviewing too

Sounds like my information is out of date!

This is pretty much -Os. Void Linux provides -O2 builds of GCC for both musl and glibc with a negligible performance difference. I have not benchmarked it lately, but it is clearly below 10%.

Anecdotal but I work on a performance oriented application and there is no such measured performance penalty from musl. For that particular use-case, like all benchmark caveats. Older musl, perhaps. Musl in other domains that bottleneck on something less performant, perhaps. Ymmv, measure twice cut once. :)

I went from an Ubuntu to an Alpine desktop and it was like bolting on a turbo. You know that rush when the turbo spools up and then it just plants you in your seat and you rocket away? Like this wasn't just "I think it's faster", I was like laughing and shouting it was so much faster. It doesn't seem quite as fast now; I feel like I must have added on enough extra software and crap now that it's close to where it was before. But the feeling when bringing up the desktop for the first time was visceral.

A nice feature of Alpine Linux is that the package repository is a relatively straightforwardly-structured git repository containing build scripts for each package:

https://gitlab.alpinelinux.org/alpine/aports/

It's worth reading the contributor guide[1] before signing up and providing any fixes and improvements - and from there, hopefully it's a relatively familiar workflow for many developers.

[1] - https://wiki.alpinelinux.org/wiki/Alpine_Linux:Contribute

I use it mostly as a build image in order to produce completely static binaries which isn't possible with most other distros because the glibc maintainers at some point in the distant past decided to impose their opinion on the matter upon you by making it impossible to statically link against glibc.

It seems though that glibc was inspired by musl to move away from many small shared libraries to a single one, namely the dynamic linker executable also being lib{c,m,...}.

It's not at the same level, but it's progress...

I really like statically linking everything. I think the trade off space is a lot different now though. But I still agree they shouldn’t have done that

I've done the same alpine trick for static binaries but may I introduce you to musl-cross-make?

https://github.com/richfelker/musl-cross-make

Just burned out static toolchains that make me static binaries for all architectures gcc supports. Much like musl.cc but they suggest building your own and I do.

I use these toolchains on debian (/ anywhere a non-ancient linux kernel runs) to make static binaries, you can too!

When I looked into this, I decided against musl-cross-make because all the tools in there are way out of date. Due to its use of custom patches, you can't easily update it yourself; you must use their specific outdated versions. They're a full major version behind on gcc (it was two major versions behind at the time), and they're two minor versions behind on musl itself. Binutils is five minor versions behind. It was much easier to just fire up an Alpine build container with the latest versions of everything, and I'm actually capable of keeping it up to date. The Alpine folks seem to have more manpower at their disposal for keeping the packages up to date with upstream.

Not an apologist by any means and all that makes sense. I've found it easy and not-a-problem to bump those versions in the config file and made gcc 11.2.0 compilers with everything up-to-date this way quite recently.

There's also this fork of MCM that's a fair amount more active, as it backs musl.cc. I probably should have mentioned it

https://git.zv.io/toolchains/musl-cross-make

Regardless, your argument that a distro taking care of something for you is exactly what they're there for. Makes sense to me.

Mostly surfacing that there's multiple ways to handle that sort of thing. I find cross compiling simplest with these and version bumping wasn't a big deal at all compared to the complexities of other ways of cross compiling.

I don't think I had seen that fork before, thanks for the link. They appear to be much more up-to-date. There are still issues using Alpine as a development container. I ultimately resorted to a separate Ubuntu container for development, building against glibc. It was the only way to get VSCode's C++ and debugger extensions to work on aarch64. I don't love the workaround; using Ubuntu for everything would be nicer. I only use Alpine for test and release builds.

Absolutely beautiful little OS - I love that it manages to balance well between being light, secure and rather versatile for server environments. I run it as a WireGuard & Docker server on a low-end VPS, and it just works without any problems at all thus far.

Its wiki does a fine job at being a pretty good resource for documentation, too, and both ArchWiki & Gentoo Wiki work well as fallbacks.

Great for containers, embedded devices, and servers.

Not quite ready for desktop development use yet despite my best efforts.

I use Arch btw.

I seem to recall some pretty poor performance regressions in Docker containers versus equivalent Debian-based container configs.

Using musl instead of glibc leads to enough surprises that I've migrated my alpine containers over to Debian-slim based ones. If you have a bunch of containers built on the same base, the storage of Alpine vs Debian becomes less substantial, and you avoid a lot of weird bugs and a surprisingly large performance hit (at least for python)

Started using Alpine as a curiosity for embedded/high security development a while ago, probably 2011-2012. I've of course used it a fair bit in $day_job containerization of services. Nowadays personally I find myself using it for the bare metal OS on servers running minimal services: things like VM hosts, purpose-built fileservers, etc.

It's a fine distro, you can typically go a good bit of time between updates without worrying about fragging your install, which is especially important to me on personal projects, which are often more like pets than livestock :P

Question, we're a long time Windows shop and just recently have been spinning up some linux containers (redis, rabbitmq). The decision fell to me and I went with Ubuntu. (I'm confident in our networking team that everything is firewalled well and not externally accessible so these are local LAN only.)

Should I instead be using Alpine? If someone is isn't distro aficionado and is utilizing "mainstream" docker images, what's the recommendation?

Ubuntu will likely be more compatible and easier to work with. The version you get in docker is very minimal as well, although maybe not quite as tiny as alpine.

My understanding of Alpine is that it is a minimal distro more suited to embedded systems and similar (happy to be corrected on that).

If you're using headless boxes for servers then Ubuntu or the house variety of the cloud platform of your choice is almost certainly best.

If you're looking for employee workstations I would go for something that looks and feels like Windows like Linux Mint or Kubuntu.

"local LAN" = local Local ....

For shame, but I'm leaving my comment as is

The maturity of your organisation’s use of containers is quite important: Alpine is a fantastic platform to build very lightweight images, but there’s organisations where people want containers that have everything. I’d say, as your usage of and experience with containers grows, aim to build more lightweight images and eventually you’ll get to the point where Alpine is the logical next step, rather than being the right choice because people on HN love it.

I think Ubuntu server is a fine choice for your needs. I tend to use it on small VPSes as it's easy to setup and get running, relatively up to date and in wide usage.

Distros seem to be having a hard time updating glibc, and there are known vulnerabilities with the pinned version of most of them. Musl/Alpine doesn't have that problem, because it doesn't use glibc.

Ubuntu comes with snaps. If you don't use snaps, they are consuming a not insignificant portion of boot time. Use Debian if you need something glib-based. If you must use Ubuntu, make sure you are using slim: you don't need manpages etc. in a production container.

Overall boot time is affected by image size, because your cluster has to download that image from $SERVER. Alpine is smaller than most, scratch is even better (if you can pull it off). Building containers with Nix can result in stupidly small images.

Its not clear-cut, but hopefully that helps you make a choice.

> just recently have been spinning up some linux containers (redis, rabbitmq)

Not sure if you're trying to make your own containers or not, but stick to the stock Docker Hub containers as much as possible. If there's an "official" Redis container, use that, regardless of the base image. Since it's supported, other people will be fixing bugs and upgrading things so you don't have to.

> If someone is isn't distro aficionado and is utilizing "mainstream" docker images, what's the recommendation?

For non-container stuff, CentOS is the "enterprisey" distro that's safest in case you one day have to support an app that only supports RHEL (and it's the base for Amazon Linux, in case you use AWS).

For container stuff, it's a toss-up between Ubuntu and Debian; I'd use Ubuntu for developer-oriented stuff, and Debian for anything else. You can of course use Alpine for containers, but there are some maintenance/development costs that come as well, so I only recommend Alpine if you really need to save space.

I recently deployed a small fleet of Dell thin clients around my house to run zwavejs2mqtt[1] via docker. I decided to use Alpine because the machines have 2GB of ram and 8GB of eMMC storage and I figured Alpine would minimize resource usage. They're working great so far!

[1]: https://github.com/zwave-js/zwavejs2mqtt

I'm typing this from an Alpine laptop. It still has a ways to go in terms of being as user-friendly as Slackware, but it's getting there. Some trouble at the moment is figuring out all the packages you need to install (and how to configure them) to get an actual working desktop as you'd expect it. There are also bugs in things like Bluetooth support and some other things. But overall it's quite usable, and very fast. You end up using Flatpak for any proprietary apps once dealing with glibc compatibility problems becomes too annoying. Some apps on Flatpak don't really work, like Slack (maybe it's possible to get it to work, but after a few days I gave up). Electron apps in general are a huge pain. I've gotten 1Password to work in Docker after over a week of futzing around. You really learn how much of a nightmare the Linux desktop today is. You also learn how much of the desktop (and non-desktop) apps actually require systemd to run at all, and how people have had to come up with fake systemd components. You definitely learn a lot about Linux!

>due to the great popularity of Docker, Alpine Linux is one of the most deployed operating systems currently in use

Wow, that's impressive! However, when I check on Distrowatch it's only in the 30s. Is there a better statistic? Something more representative of actual amounts of installs on anything at all?

I've been down this path before and I can say for sure two things -

1 - Distrowatch stats are both gamed and useless

2 - There's no easy way to see which distro is most popular on the whole.

DistroWatch counts by clicks to the distro's page from what I remember. Its statistics represent the "interest", rather than install count.

With regard to Alpine Linux on Docker, the install count[0] of the base image should give an idea (over 1 billion!).

[0] https://hub.docker.com/_/alpine

Edit: clarity

iirc every intel cpu runs minix on its management engine. Might be a contender.

I'm a big fan of Alpine Linux, and use it as a daily driver.

Here is my Alpine Linux minimalist desktop setup.

https://github.com/git-sgmoore/AlpineLinux-DailyDriverDeskto...

It’s used to build pretty much every Docker image around us. Works extremely well as WSL. Super lightweight, performant, and the number of packages is growing strong.

Pair it with xcfe (or gnome or kde) and you have a super desktop, no more worries about RAM or background processes eating up CPU (looking at you Windows).

Yes, there’s a Blender package! and many other good stuff.

Alpine made Linux fresh again, and yes I have 32GB of RAM and a fast CPU and SSD, using Alpine will make sure it stays fast

It needs much less than that to run, of course, it’s brilliantly focused.

depends on what you do, it's musl-based, some glibc software might not run well, and if you use linux for daily development work, it might not fit.

I usually (mostly) use containers for dev, and deployment, which abstracts things a bit

> performant

Isn't musl generally less performant than glibc?

I think busybox is lighter, maybe not faster, but make alpine probably the cleanest modern distro, make it efficient and fast in the process.

I don't think busybox has anything to do with the question of musl performance.

Alpine was the first of its kind but there are better minimal distributions now.

I’d rather not tip toe around the oddities that alpine introduces and just use a Debian based image.

What are those better distros and why are they better?

debian-slim and its downstream, because fighting with musl libc isn't worth the time.

I'm working on a personal project where musl is the foundation of my project. So I'm interested in learning about all the downsides, corner cases, and footguns of using musl. Can you please share some pointers (search phrases, mailing list/forum posts, etc.) for me to read more.

I saw a thread the other day on docker command pull alpine:latest is not good. Im pretty sure like every enterprise docker container is running this command. Anyone see anyone change this or bring this up at work?

What do you mean, "docker command pull alpine:latest is not good"? Was there a specific breakage?

They’re probably referring to the “latest” tag. The “latest” tag is a terrible idea to be anywhere near anything that matters but I’d be very surprised if any enterprise is depending on a latest tag.

Digest pinning is what should be used in any system that matters.

About musl and performance: I have noticed that ram usage is noticeably lower.

What are the chances of encountering performance issues and/or incompatibilities when using Alpine Linux as Docker image for Go apps in production?

Aren't go apps statically linked? You could theoretically build it and put the binary on a "FROM scratch" docker and run it.

I mean, what I'm trying to say is that the base image doesn't matter, it would be basically just a debugging environment for you to exec into.

brilliant except for weird edge case failure modes that aren’t obvious, like random dns lookup failures or random network dropouts.

postmarketOS is based on Alpine Linux and it's the perfect choice for a lightweight mobile OS.

Worrying about linux distros seems so antiquated in this serverless cloud computing world. Or at least so last decade.

While the difference between some distros is almost academic, other distros have very strong principles that lead them to having very different forms and functions. Alpine falls into that category - if one were to not 'worry' about distros as you seem to be suggesting and downloaded Alpine without reading up on it, you would likely be put out by what you find.

Alpine is a distro with a purpose, and that is worth 'worrying' about.

You say that but there's other reasons to use linux (on the desktop!), and serverless itself is an old concept in which I never used to care which flavour of unix/linux my host was running my php or cgi scripts on.

Reply on news.ycombinator.com