Hacker News Reader

My biggest beef with the apt/deb-based distros for container images is that apt-get updates installs take frustratingly long, whereas apks always tend to be near instant. I wonder what it is in the postinst and related scripts that just take so long and can't be parallelized!

Most of the reason I switched from Ubuntu -> Arch is that working with alpine allowed me to realize that installing packages and their dependencies don't have to take so long.

Have you tried building the images with eatmydata library?

Essentially, dpkg is hilariously slow due to its approach in issuing fsync calls for every file - eatmydata wrapper injects a shared library which turns that into a no-op.

I haven't timed it rigorously, but I experienced up to 10x speedup in container builds by just wrapping dpkg with eatmydata.

[Edit: modern debian images have “force-unsafe-io” turned on in /etc/dpkg/ so eatmydata shouldn’t make a difference.]

I’m not the OP, but in an example I just tried it took build-essential from 48s to 39s (timed on the third step, without and then with the call to eatmydata):

  from debian:sid
  run apt update && env DEBIAN_FRONTEND=noninteractive apt-get install --yes eatmydata
  run apt update && eatmydata env DEBIAN_FRONTEND=noninteractive apt-get install --yes build-essential

Can you replicate something faster? Was I doing something wrong?

Yes, the second invocation will have hot caches. Basically the standard is to either test in a clean environment (e.g. immediately after a reboot) or to run the command until the times stabilize, and then take, say, the middle 3 out of 5 measurements.

What is cached though? DNS? IP routes to the Debian mirrors?

I ran it three more times. Plain install was 50s 48s 47s; eatmydata was 46s 46s 46s.

Installing build-essential probably does enough non-dpkg IO to account for the difference (install scripts etc, for example when libc changes and the system scans for running daemons) but it isn’t much and the delta may well be statistically insignificant.

File system. Drives cache recently read blocks. This isn’t something that’s neesesarily e en visible to the OS.

I decided to benchmark these scenarios out of curiosity, because I hadn't heard of eatmydata before or of force-unsafe-io.

- Docker image: `debian:stable-slim`

- `docker system prune -a` ran before each test.

- Packages installed: `build-essential ca-certificates libssl-dev software-properties-common wget curl git`

Results:

Default Debian settings: 48.32 seconds

force-unsafe-io enabled: 55.86 seconds

eatMyData enabled: 35.04 seconds

It's possible that the last time I dealt with that on debian force-unsafe-io wasn't set in /etc/dpkg.

I haven't really built debian-based containers in last two years.

This here. Honestly most orgs with uhh.. Let's say a more mature sense of ROI tradeoffs were doing this from pretty much the very beginning.

Also, Ubuntu 22.04 is only 28.17MB compressed right now so it looks equiv to debian-slim. There are also these new image lines, I can't recall the funky name for them, that are even smaller.

I'm pushing to go back to Debian from Ubuntu. Canonical's making decisions lately that don't appeal to me, and especially on the server I don't see a clear advantage for Ubuntu vs good ol' rock solid Debian.

Host OS vs container image base shouldn't be the same conversation.

I've done this multiple times when new Debian Stable was released (Bookworm just got released). Its great, until it isn't (ie. you need more up-to-date software). Although to be fair, you might be comparing Ubuntu LTS with Debian releases and the minor Ubuntu releases get short support. Ubuntu is akin to a polished Debian Testing snapshot. I don't like them swapping to Snap either (snappy it ain't).

why is it so much smaller than debian slim?

It's not, Debian slim is ~28.8MB compressed.

If you want to compare compressed size, then alpine is 3.25MB

The parent comment was comparing Ubuntu and Debian-slim

Fewer features (including less "we've supported doing that for 20 years and we're not cutting it now"), packages separated into parts (ex. where debian might separate "foo" into "foo" for the main package and "foo-dev" for headers, alpine will also break out "foo-doc" with manpages and such), general emphasis on being small rather than full-featured.

> There are also these new image lines, I can't recall the funky name for them, that are even smaller.

You might be thinking of the chiselled images. An interesting idea but very much incomplete[1].

[1]: https://github.com/canonical/chisel-releases/issues/34

Yes, that's it! I've been working with .Net a lot recently and recalled Microsoft's announcement about them. Specialized runtime images for JRE, .NEt, Asp.Net, and etc.

In the same boat here as well. Especially when you’re talking about container images using JavaScript or other interpreted languages that are bundling in a bunch of other dependencies, the developer experience is much better in my experience given that more developers are likely to have had experience working in a Debian based distro than an Alpine based one.

Especially when you’re also developing within the container as well, having that be unified is absolutely worth the convenience, and honestly security and reliability as well. I realize that a container with less installed on it is inherently more secure, but if the only people who are familiar with the system are a small infrastructure/platform/ops type of team, things are more likely to get missed.

Can you point me on where to look for more details on securing a container? I'm a developer myself, and for me, the main benefit of containers is being able to deploy the app myself easily because I can bundle all the dependencies.

What would you suggest I restrict at runtime and can you point me to a tutorial or an article where I can go have a deeper read on how to do it?

What you want to read is the kubernetes pod security context fields[1].

In your Dockerfile, add a non-root user with UID and GID 1000, then in the end of your Dockerfile, right before a CMD or ENTRYPOINT you change to that user.

In your kubernetes yaml manifest, you can now set runAsNonRoot to true and runAsUser & runAsGroup to 1000.

Then there’s the privileged and allowPrivilegeEscalation fields, these can nearly always be set to false unless you actually need the extra privileges (such as using a GPU) on a shared node.

Then there’s seccomp profiles and the system capabilities. If you can run your container as the non-root user you’ve created, and you don’t need the extra privileges then these can safely also be set to the most restricted. Non-privileged non-root is the same as all capabilities are dropped.

The tricky one is the readOnlyRootfilesystem field. This includes /tmp, which is considered a global writeable directory, so the workaround is to make a in-memory volume and mount it at /tmp to make it writable. Likewise, your $HOME/.cache and $HOME/.local directories (for the user you created in your Dockerfile) are usually used by third party packages, so creating mounts here can be useful as well (if for some reason you can’t point it to /tmp instead).

[1] https://kubernetes.io/docs/tasks/configure-pod-container/sec...

> In your Dockerfile, add a non-root user with UID and GID 1000, then in the end of your Dockerfile, right before a CMD or ENTRYPOINT you change to that user.

I'm not certain, but would it be more secure to create a non-root user in the Dockerfile, but _don't_ use UID 1000. Use volume mounts to grant host disk access instead.

If you want to use the runAsUser and runAsGroup options in kubernetes that user needs to exist, hence why creating it is needed.

In our case, we have our own base container images that include this user. The base images are either just plain copies of official ones (such as dotnet or curl) or also includes stuff that almost everyone needs (such as MSSQL ODBC drivers, libxml2, curl etc) in the case of python/R/node.

This was really helpful for me:

https://cheatsheetseries.owasp.org/cheatsheets/Docker_Securi...

Basically you need to work as unprivileged user and with immutable file system (of course you can have ephemeral /tmp or persistent /data, but generally the entire system should be treated as read-only).

You might want to check out Wolfi and Chainguard Images. Wolfi is a Linux distro that we use to build minimal images that are roughly comparable to Alpine in size but, everything is compiled from source against glibc.

Our images come without a shell or pacakge manager by default, but there are -dev variants that include these.

https://github.com/wolfi-dev/ https://github.com/chainguard-images/images

I was using wolfi for an oss project, but the recent attempts at forcing payments when pinning version was a huge red flag, as was the messaging around it, and I’ll be migrating away from it.

https://www.chainguard.dev/unchained/scaling-chainguard-imag...

Pinning a language version (say python 3.11) isn’t an optional thing its a best practice, and the notion that its because of security seems intentionally misleading as the images should be refreshed in place on the tag along with signatures.

I'm very sorry that we broke things for you.

To be clear, nothing has changed with Wolfi. Wolfi is an open source community project and everything is still available there: https://github.com/wolfi-dev/.

We have made changes to Chainguard Images - our commercial product built on top of Wolfi - which mean you can no longer pull images by tag (other than latest). Chainguard images are rebuilt everyday and have a not inconsiderable maintenance cost (and the money we make here directly helps us support Wolfi).

The easiest way to avoid this is to build the images yourself. You can rebuild identical images to ours using apko and the source files in the images repo e.g: https://github.com/chainguard-images/images/blob/main/images... (note you can replace package names with versioned versions). You can also just use a Dockerfile with the wolfi-base image to "apk add" packages. Full details are here: https://www.chainguard.dev/unchained/a-guide-on-how-to-use-c...

I agree that pinning is a best practice. The above blog explains that you can still do it using a digest, but I accept this isn't the simplest solution.

If I can help any more, please feel free to get in touch - you can find me most places including twitter https://twitter.com/adrianmouat

I sympathize, but if wolfi is oss, please update docs and downloads to use a separate registry then your commercial one for distribution.

else, frankly you claimed wolfi is oss, get customers to use your registry, and then bait and switched your early adopters.

aka, major version upgrades at random, have fun!

Wolfi packages are served from https://packages.wolfi.dev/os which can be used with apk tools or apko.

The built Chainguard images are all on the cgr.dev registry.

Wolfi is completely OSS.

The policies regarding Chainguard Images have changed over time, so if there are docs that don't properly reflect this, please let me know and I'll get them updated.

let me rephrase that, wolfi is an oss container image distro without an image, you get to pay us for the image. but here's a our tooling cause we like to call it oss, have fun building. either have those oss images on a non commercial registry, or let's stop misleading folks about what this is, aka pay to play/use oci images. that all wolfi docs / blog posts reference a commercial registry while touting it as oss, is the definition of bait and switch.

> the difference between alpine (~30 MB) and debian-slim (~80)

Given that it's a different layer, the your container runtime isn't going to redownload the layer anyway, right?

Exactly. Part of the appeal to consolidate all of our container images to use Debian-slim is the ability to optimise the caching of layers, both in our container registry but also on our kubernetes cluster’s nodes (which can be done in a consistent manner with kube-fledged[1]).

[1] https://github.com/senthilrch/kube-fledged

Thanks for that - that operator sounds extremely useful!

And even if it did, in an ancient data center that only uses gigabit ethernet, that's only a .5s longer download. And even a $4 DigitalOcean server comes with 10GB of storage, so that 50MB is only 1/200th of the instance's store. (I'd also bet that nearly no one uses instances that tiny for durable production work where 50MB is going to make a difference.)

I've run into the same thing for large dev images, but using pure rust often means that musl allows for a single executable and a config file in a from scratch container for deployment. In cases where a slab or bump allocator are used, musl's deficiencies seem minimized.

That means duplication of musl in lots of containers, but when they are all less than 10MB its less of an issue. Linking against gnu libraries might get the same executable down to less than 2MB but you'll add that back and more for even the tiniest gnu nase images.

what is pure rust,rust needs its own libraries which is a few MB as I recall

By pure, I just meant no dependencies on c or c++ bindings other than libc. If that is the case you can do a musl build that has no dynamic dependencies, as all rust dependencies are static. So then your only dependency is the kernel, which is provided via podman/docker. A decent sized rust program with hundreds of dependencies I can get to compile down to 1.5MB. But that is depending on gnu. So if you had 4 or 5 of those on a node, it might be less data to use one gnu base image that is really small like rhel micro, and build rust for gnu. But if you have cpu hungry services like I do, then you usually have only a couple per node, so from scratch musl can be a bit smaller.

> cpu hungry services

Have you benchmarked musl vs glibc in any way? Data I've seen is all over the place and in curious about your experience.

Very few system calls in the main loop. Not even close to being io bound, and as I said memory is all preallocated, so allocator efficiency isn't a factor.

Edit: realized I didn't answer your question, I ran flamegraph both ways and it was completely dominated by tight vectorized loops and a couple sadly necessary memory copies.

I made the switch too around 4ish years ago. It has worked out nicely and I have no intention on moving away from Debian Slim. Everything "just works" and you get to re-use any previous Debian knowledge you may have picked up before using Docker.

Do you have any tips regarding building R-based container images?

R is kinda difficult and I haven’t cracked this one. Currently we’re using the rocker based ones[1] but they are based on Ubuntu and include a lot of stuff we don’t need at runtime. I’ll look into creating a more minimal R base images that’s based on Debian-slim.

[1] https://github.com/rocker-org/rocker-versioned2