Hacker News Reader

Comment by jrm4 | Hacker News Reader

Am as skeptical of this as I am of 'serverless' which also doesn't mean much and isn't true.

What does this actually mean and why do they hide it behind a nonsensical term?

I haven't seen ubuntu's version of distroless but the general term implies a container image with an empty shell of what a linux distro provides. Leaving behind only the minimum files for running your app rather than all the cruft around running and maintaining a distro, which is normally all done at container build time.

From the Debian based https://github.com/GoogleContainerTools/distroless/blob/main...

Statically compiled applications (Go) that do not require libc can use the gcr.io/distroless/static image, which contains:

    - ca-certificates
    - A /etc/passwd entry for a root user
    - A /tmp directory
    - tzdata

Applications that require libc but do not need libssl can use the gcr.io/distroless/base-nossl, which contains all of the packages in gcr.io/distroless/static, and

    - glibc

Most other applications (and Go apps that require libc/cgo) should start with gcr.io/distroless/base, which contains all of the packages in gcr.io/distroless/static, and

    - glibc
    - libssl

What I have found is that at least Google's distroless images do not get updated frequently enough to handle security issues.

Because of this, I ended up switching to Ubuntu images and doing apt-get dist-upgrade before anything else to try to pick up fixed binaries.

If Ubuntu truly is able to keep an up to date cadence for its distroless images, then I'd definitely want to investigate to see if I should switch.

Looking at the dependency listing I'm trying to figure out why this would be. There are like 4 dependencies here in the largest of these containers, so why is it so difficult to keep them up to date?

The issues I've seen are around updating the run times to point to new release versions in a timely manner. The project maintainers don't actively work on (the endless) security updates and defer people to commercial projects that can spend the time doing that.

The images are tiny, and trivial to update. You don’t really need theirs, just take the bazel rules and build it yourself.

I'm using rules_distroless+rules_oci with bazel to build small Ubuntu based images. It takes some effort, but ends up being pretty nice.

It means that your containers only have what you explicitly put there. There's none of the niceties that the distro's typically include.

It means you have to stop treating your containers a little less like VM's and start treating them a bit more like OS processes. For instance, you can't get a shell on one unless you bother installing that shell yourself.

How would you install a shell without a package manager?

Edit: @mhio That's what I suspected, thank you.

I assume the images are pre built based on a manifest provided by the developer. The developer decides whether or not to include a shell and which shell to include.

What are these manifests? I presume you are correct - the images are prebuilt (as typical for docker), but then it's like coreos, what you get is whatever you `docker pull'ed, most likely very basic (by design).

Great in cases where the app is a self-contained Go program only requiring libc, but not useful when you need to apt/yum/dnf install anything.

You don't.

There are tools like `nsenter` to act on the container namespaces.

https://manpages.debian.org/bookworm/util-linux/nsenter.1.en...

Is this that hard to do on your own? You can base from scratch if you want. Then you’d have to add just what you need which I guess could be hard.

Probably not, I've never tried it. But I assume you can waste a bit of time puzzling out the few things to include which are actually needed every time, and a distroless image has that pre-puzzled for you.

If I wanted to be so specific about what's going into my images I'd probably just build them with nix.

I guess the key thing is that they promise the security fixes for 12 years.

This is interesting, because in large org you ofren have lot of stuff you just want to keep on running with minimal changes.

I would say a lot of developers don't really know their way around a Linux distro as well as you might think?

It's the same logic as frameworks or other building blocks: why leave all these decisions up to individual teams, when everyone should be doing things more or less the same way?

If need to do something custom, you'll know what you're doing and you can do that. Otherwise, use good defaults provided by people who know better than you.

I mean... one certainly could do this. And then as soon as anything breaks with that container, the person trying to fix it will curse the day you were born. People already go way too far with stripping container images, and it's awful for troubleshooting. The last thing we need as an industry is taking that silly trend even bother.

If you're going to use minimal container images you definitely need to know how to inject tools into the container.

Yeah and it's a true pain. Double that if your container security policy requires running in fully rootless mode (i.e. not just the usual CAP_ALL drop but also container uid != 0), quadruple if you're in a Kubernetes environment, and even more if you can't access the nodes themselves directly (AWS EKS).

I'd say you need to know how to properly instrument your containers so that you'll never need to get inside. If your team is on the level of "we need tools inside the container for troubleshooting", then don't use minimized containers like these.

It's about security and reducing the probability of getting CVEs for your running containers. Less software you bundle there, less security issues you will have. And this in turn means less work on maintenance as you don't need to update, test and deploy new versions of images so often.

When the cow gets sick, you shoot it.

You still want to know why the cow got sick, in order to prevent all other cows in the stable from falling sick as well.

And ideally, you want to keep the cow alive, at least if you fed it artificially.

You're not looking at this from a DevOps perspective.

In real-world cloud environments, logging into the container and expecting to have a complete Unix environment to poke around in just isn't an option. You might not even have the required permissions to do so, as a matter of corporate policy. You need to build observability directly into the application -- or you haven't built a complete cloud application.

> You need to build observability directly into the application

This is not objectively better than external observability.

If the "DevOps perspective" requires it, then it's a pointlessly inflexible perspective.

> just isn't an option

You don't get to blame the system when you made the system.

The option is there.

That's the key thing why DevOps is frankly a pile of bullshit and why so many companies ridiculously overpay on cloud infrastructure.

When you got clueless morons who were in a 6 week nodejs bootcamp now "developing" containers, with no one even having the permissions to troubleshoot crashes - that may very well originate from someone trying to get remote code execution - it is absolutely no surprise that hacks, ransomware, infrastructure so brittle that it only doesn't fail because EBS / EKS keeps restarting the pile of dung every minute and infrastructure pieced together from 40 AWS services where one classic EC2 VM would be sufficient are plentiful.

Developers having shell access to production container is not the best practice, to say the least. For debugging locally, it's very easy to nsenter your container.

We use Google's distroless containers for production, but we also have alternate Dockerfiles available in our CI/CD for building a more traditional container that can be logged into if there are application issues to debug.

It's not an exact copy of prod, but so far (a few years, and thousands of containers) it hasn't really been an issue as far as I've heard. We don't do anything fancy though. Node, Go, Java and Python apps..

I think the point is that you should have a plan (which may or may not require debug tooling in the image) and you should include what you need for that plan and nothing more.

Finding something useful that the distro left lying around but that wasn't part of the plan might feel like a win here and there, it's just as likely to be a win for your adversary as it is for your friend.

They are not hiding behind a non sensical term. Distroless is a established term in the container space. Many images are often offered in both "normal" and "distroless" flavor, where distroless is stripped of anything that is not needed to run the application, for example curl and shells etc.

Since distroless are lacking all these tools other techniques needs to be used during troubleshooting and debugging.

I think distroless images are great. Less stuff to patch and smaller images.

A term can be both established and nonsensical, like grandparent's "serverless".

It doesn't make sense in multiple ways.

"Distro" means "distribution", so "something that is distributed". Aren't "distroless" images being widely distributed?

It seems that "distroless" images are being built from a distribution, and then by removing some of the contents. So you must have a distro first, otherwise you can't have "distroless". That doesn't add up semantically.

Words can have different meaning than its etymological parents.

If I seed Ubuntu it doesn't become blckoil distro. So the word distro means different from distribution.

The issue is you could make and maintain a blckoil distro tomorrow that only keeps the bare minimum to run a go application. It would be bit for bit the same as the "distroless" version, except it will be a distro.

The "something that is distributed" is a package manager and package repositories. What makes a distribution is what choices it makes in letting you install software.

A distroless image doesn't include a package manager. You cannot use anything in the provided distroless image to install software.

That doesn't make it not a distro, that makes it a different kind of a distro. I'm not sure if we've had ones that absolutely had no package manager, but we've definitely had TONS for which the intent ISN'T installing extra/different software,e.g. a bunch of the specialized raspberry pi ones.

Do you think Linux From Scratch https://www.linuxfromscratch.org/ is a distribution? That project is not a distribution, it helps you build your own distribution. The Linux kernel by itself, although the source and binaries are "distributed" to you (i.e. English-dictionary definition of distribution), is not a distribution.

There is no essential difference that makes Linux From Scratch not a distribution. None.

Some distros come with a lot of stuff. Some with very little.

That it can be used to make another distro has NO BEARING on this. What would you call that thing if it's not a distro? What is that noun?

> A term can be both established and nonsensical

Not at all. The establishment of a term, and giving it a definition that's widely agreed upon, by definition (heh) makes it not nonsensical.

> "Distro" means "distribution", so "something that is distributed"

No. In the Linux world a distribution means something much more restricted than that. It's a full system that can be installed, complete with a package manager, init system, etc.

"Distroless" simply means it's a minimal set of software and libraries such that it can run a particular binary that you copy into it, but it doesn't act like a distribution. You can't install libraries and programs into the container using a package manager, for example, like you'd be able to if your container held a stock Debian install.

> It seems that "distroless" images are being built from a distribution, and then by removing some of the contents. So you must have a distro first, otherwise you can't have "distroless". That doesn't add up semantically.

I don't see where you're getting that conclusion from. If you take a distro, install it into a container, and then strip away everything that makes it a distro, then it is indeed distroless. How you got there is irrelevant.

You could even build a distroless container without installing the distro into the container first. You could install the distro into a chroot (or into another container) and then selectively copy things out into the target container. Or if you were a masochist, I suppose you could even download source tarballs of everything you wanted in your distroless container, and manually build them on a host system and copy them into the container.

Again, how you get to the final state is irrelevant. The idea is that the final state is runnable, but does not contain all the bits you'd expect to see in an installed distro.

I've seen nothing to indicate that your definition of "distro" is THAT restrictive.

I'm confident saying a linux distro is anything with a linux kernel that you can slap on bare metal or a VM system and get something up and running.

You're merely describing a minimalist/specialized distro, a not particularly new or unique concept.

I mean in the beginning you'd be downloading the kernel code yourself and building a distribution with core utils. Distros came along and did all that work for you: they were a system that packaged up the kernel with a lot of useful utilities as well as an packaging and update mechanism.

Of course you can find a way to be upset with the term if you want to, but I think it's in this sense that these container images are distroless.

The term makes a lot of sense when you read it as having the package manager at it's core, which also happens to be the update mechanism. But in the container world, that would be an idle update mechanism and package management (as in selecting the subset of installed stuff) is also handled through image selection. But an image created from scratch by whoever would be considered end user in the container world would still contain the package manager of the distribution used to set up whatever is eventually imaged. With this as the baseline, distributionless is either an image that has this umbilical removed, or, as purists would probably prefer, an image where the package manager has never existed because everything was set up without involving one, or set up from the outside, without ever running anything "inside".

So they could have called it "minimalist", "barebone", "stripped".

I share parent's view that it's marketed with the most provocative term to attract attention, where an appropriate name wouldn't have stirred enough controversy to their taste ?

I'm quite surprised by this descriptive term also; in fact I've always (for years) called this type an "application container" since it only contains the files required by the primary process and doesn't have a standard init system (usually it is expected to handle PID 1 responsibilities in its primary executable or use a stub init (as when using e.g: systemd-nspawn --as-pid2 ... ).

In contrast a "system container" has a regular init system and likely also package management (although that is optional) but was created using a package manager so the miscellaneous package-installed files are usually in there.

They could, but "distroless" is already an estabilished name, I think pioneered by Google[1]. As someone close to this field, I immediately know what "distroless" means in this context, and it's unclear what "minimalist" mean (does it mean it's based on Ubuntu Minimal? Alpine? Distroless?).

[1]https://github.com/GoogleContainerTools/distroless

those terms have already been used in various distro installers for installations that contain very little, but still all the setup for you to expand it into a full blown distro with the package manager.

The distroless images specifically don't give you a package manager, only specific packages needed for you to run.

Think in the way that even without kernel and toolkit, many deployments still need libraries and frameworks. Like python deployment needs python runtime, AI deployment needs Python and pyTorch.

Canonical will support that.

It's actually make sense as you don't need package manager or other things (basically distro toolkit) to build and run container image.

Reply on news.ycombinator.com