Hacker News Reader

Comment by __MatrixMan__ | Hacker News Reader

It means that your containers only have what you explicitly put there. There's none of the niceties that the distro's typically include.

It means you have to stop treating your containers a little less like VM's and start treating them a bit more like OS processes. For instance, you can't get a shell on one unless you bother installing that shell yourself.

View on HN

I mean... one certainly could do this. And then as soon as anything breaks with that container, the person trying to fix it will curse the day you were born. People already go way too far with stripping container images, and it's awful for troubleshooting. The last thing we need as an industry is taking that silly trend even bother.

If you're going to use minimal container images you definitely need to know how to inject tools into the container.

Yeah and it's a true pain. Double that if your container security policy requires running in fully rootless mode (i.e. not just the usual CAP_ALL drop but also container uid != 0), quadruple if you're in a Kubernetes environment, and even more if you can't access the nodes themselves directly (AWS EKS).

I'd say you need to know how to properly instrument your containers so that you'll never need to get inside. If your team is on the level of "we need tools inside the container for troubleshooting", then don't use minimized containers like these.

It's about security and reducing the probability of getting CVEs for your running containers. Less software you bundle there, less security issues you will have. And this in turn means less work on maintenance as you don't need to update, test and deploy new versions of images so often.

When the cow gets sick, you shoot it.

You still want to know why the cow got sick, in order to prevent all other cows in the stable from falling sick as well.

And ideally, you want to keep the cow alive, at least if you fed it artificially.

You're not looking at this from a DevOps perspective.

In real-world cloud environments, logging into the container and expecting to have a complete Unix environment to poke around in just isn't an option. You might not even have the required permissions to do so, as a matter of corporate policy. You need to build observability directly into the application -- or you haven't built a complete cloud application.

> You need to build observability directly into the application

This is not objectively better than external observability.

If the "DevOps perspective" requires it, then it's a pointlessly inflexible perspective.

> just isn't an option

You don't get to blame the system when you made the system.

The option is there.

That's the key thing why DevOps is frankly a pile of bullshit and why so many companies ridiculously overpay on cloud infrastructure.

When you got clueless morons who were in a 6 week nodejs bootcamp now "developing" containers, with no one even having the permissions to troubleshoot crashes - that may very well originate from someone trying to get remote code execution - it is absolutely no surprise that hacks, ransomware, infrastructure so brittle that it only doesn't fail because EBS / EKS keeps restarting the pile of dung every minute and infrastructure pieced together from 40 AWS services where one classic EC2 VM would be sufficient are plentiful.

Developers having shell access to production container is not the best practice, to say the least. For debugging locally, it's very easy to nsenter your container.

We use Google's distroless containers for production, but we also have alternate Dockerfiles available in our CI/CD for building a more traditional container that can be logged into if there are application issues to debug.

It's not an exact copy of prod, but so far (a few years, and thousands of containers) it hasn't really been an issue as far as I've heard. We don't do anything fancy though. Node, Go, Java and Python apps..

I think the point is that you should have a plan (which may or may not require debug tooling in the image) and you should include what you need for that plan and nothing more.

Finding something useful that the distro left lying around but that wasn't part of the plan might feel like a win here and there, it's just as likely to be a win for your adversary as it is for your friend.

How would you install a shell without a package manager?

Edit: @mhio That's what I suspected, thank you.

I assume the images are pre built based on a manifest provided by the developer. The developer decides whether or not to include a shell and which shell to include.

What are these manifests? I presume you are correct - the images are prebuilt (as typical for docker), but then it's like coreos, what you get is whatever you `docker pull'ed, most likely very basic (by design).

Great in cases where the app is a self-contained Go program only requiring libc, but not useful when you need to apt/yum/dnf install anything.

You don't.

There are tools like `nsenter` to act on the container namespaces.

https://manpages.debian.org/bookworm/util-linux/nsenter.1.en...

Is this that hard to do on your own? You can base from scratch if you want. Then you’d have to add just what you need which I guess could be hard.

Probably not, I've never tried it. But I assume you can waste a bit of time puzzling out the few things to include which are actually needed every time, and a distroless image has that pre-puzzled for you.

If I wanted to be so specific about what's going into my images I'd probably just build them with nix.

I guess the key thing is that they promise the security fixes for 12 years.

This is interesting, because in large org you ofren have lot of stuff you just want to keep on running with minimal changes.

I would say a lot of developers don't really know their way around a Linux distro as well as you might think?

It's the same logic as frameworks or other building blocks: why leave all these decisions up to individual teams, when everyone should be doing things more or less the same way?

If need to do something custom, you'll know what you're doing and you can do that. Otherwise, use good defaults provided by people who know better than you.

Reply on news.ycombinator.com