Hacker News Reader

Comment by mrshu | Hacker News Reader

I ran into this while checking connectivity between containers on an internal Docker network where the image had neither curl nor wget.

The main surprise was that Bash has /dev/tcp which lets you do the equivalent of an HTTP request with a bit of shell magic, for instance:

  exec 3<>/dev/tcp/service/8642
  printf 'GET /health HTTP/1.1\r\nHost: service\r\nConnection: close\r\n\r\n' >&3
  cat <&3

Where `service` is just the hostname of whatever you’re talking to and 8642 is the port you are trying to talk HTTP to.

Pretty cool!

View on HN

It seems pretty cool, but I am wondering if there's any drawback on just using images that support curl? I can't think of any and to me it's kinda a must have, even on production images

That is indeed a solid pushback! :)

For what its worth, this container used `python:3.12.2-slim-bookworm` and I really would not expect that sort of an image to bundle `curl` -- even if it is intended for production.

Ah I see so it was basically a minimal image that bundles just python? I can see why it wouldn't bundle curl! Thought it was a custom Image for some reason, hence my original comment

Yes, a very minimal image indeed. Had it been a custom image, curl would be one of the first things I would make sure it contains :)

You can also use the sockets lib in that case, you depend on POSIX instead of Linux

This of course only supports http, not https. It's great for health checks e.g. in a docker environment. To do https, you'd have to use something like socat, but of course that doesn't use bash only.

Https is almost always terminated separately from the application code.

It's also a two line Dockerfile to add wget or curl to almost any pre-existing container image. This is a fun idea though.

I always recommend to not have any dependencies outside of the code.

So we start at compiling the codebase (Rust) against MUSL. That way we can run it with FROM scratch images.

If we need more tooling available at runtime, then we look at alpine, but still using MUSL.

If MUSL itself is proving problematic, or if some of the libraries we use need glibc then we can look at using some locked down image.

The cool part about FROM scratch images is that you'll never have to update your base image to address CVEs. Only your software and its (compiled) dependencies.

> The cool part about FROM scratch images is that you'll never have to update your base image to address CVEs. Only your software and its (compiled) dependencies.

What's the benefit really, though? If you still need to be able to rapidly deploy a new image in response to a dependency CVE, what have you gained?

You've gained that happening much less frequently. The tradeoff is making every other problem harder to diagnose.

Debug containers are a thing.

Add an ephemeral container to an already running pod, for example to add debugging utilities without restarting the pod.

https://kubernetes.io/docs/reference/kubectl/generated/kubec...

Yup! They are a good solution to the massive problem you caused for yourself by implementing a different "solution" to a non-problem.

And even that's only true if you assume kubernetes is the only place your container runs where you might want to also debug it.

You want to ship every debug utility you will need in every image? Just seems wasteful. What about 3rd party images, you will respin images just to add your preferred toolset?

Nope, not my position at all. I want to have a minimal OS environment with rudimentary tools available with zero extra friction. FROM alpine:latest adds less than 10MB and covers 95% of use cases. Typically depending on the container I will often throw in curl and some other QoL tools too.

For the rare cases where you find yourself needing to attach a debugger to your pods running in staging/prod, a debug container is absolutely the right tool to reach for.

> every debug utility you will need in every image? Just seems wasteful

How wasteful though? I have to admit, I envy the person whose codebase itself they have to support is so lean and space-conservative that the size of the gnu coreutils, curl, nano, etc., would show up as anything but a rounding error in the image size.

I see it like putting a thermometer in a turkey before I stick it in the oven. Sure, the thermometer adds thermal mass itself, making the turkey take a few seconds longer to cook, but the value of it being there is greater than the cost imposed.

If the base image I use is based on Debian, it comes with more than 15 binaries that I don't use.

But when Docker scans my image and notices that there is a CVE in one of those binaries, my image is currently out of compliance.

FROM scratch just reduces the surface.

> FROM scratch just reduces the surface.

The actual attack surface of your application? Or the attack surface of you and your team's attention from a busybody security org.

It's important not to confuse the two.

Both. Many attacks take the form of an exploit to get a shell, then using available utilities to exploit the kernel to escape to the host. If your image has neither a shell nor utilities that won't get very far.

What percentage of CVEs can be used to obtain a shell, but can't otherwise be used to obtain some other form of code execution in a distro-less container?

I haven't run any stats and am certainly not an expert but I would expect quite a few. In the one scenario you merely need to pull off an exec with a valid path. In the other you need to either write a block of memory and mark it as executable or else write your payload out to disk and mark the file executable. So it's the difference between being able to pull off a single syscall versus most likely needing arbitrary code execution.

Important to whom?

preface: I'm not asking things rhetorically, I genuinely want to learn here.

> to not have any dependencies outside of the code.

> ... FROM scratch images is that you'll never have to update your base image to address CVEs...

So a FROM scratch image, basically doesn't have things like a package manager to install things, and maybe also libraries that things like curl would depend on? Sorry for my ignorance, I've heard of FROM scratch but never tried them.

Apparently you get only an empty root directory, nothing else. https://medium.com/@fabrizio.sgura/the-forgotten-minimalist-...

There is nothing there.

If you want to run as another user, you need to manually add an /etc/group & /etc/password (or generate them in a stage before that and copy them over).

If you need ca-certificates, you need to install ca-certificates in a stage before that and copy over /etc/ssl/certs/ca-certificates.crt from that stage to your current one.

More than one ~500 employee company I've worked at has had security policies either encouraging or requiring the use of "distro-less" images - images with no OS components other than the absolute minimum required to run the application. For go binaries this meant literally nothing in the container apart from the executable.

In theory it has a couple of benefits. You don't have to re-deploy your image to patch CVE's in OS components if you don't have any OS components. And it provides some measure of defence-in-depth - one could certainly theory-craft a scenario where an attacker gains some limited control over your application and then uses some OS component to escalate.

These days if a security engineer is proposing my team adopt distro-less containers to receive these benefits, I would point out that we need to weigh them against the very real drawbacks of not having standard debugging tools available where and when we need them. And also to consider the relative impact of other defence-in-depth measures they could be pursuing instead - such as any sort of network policy to limit network traffic.

Debug containers are a thing.

Add an ephemeral container to an already running pod, for example to add debugging utilities without restarting the pod.

https://kubernetes.io/docs/reference/kubectl/generated/kubec...

> not having standard debugging tools available where and when we need them

Keeping in mind that containers are merely a bunch of namespaces, there's nothing stopping you from entering the same PID namespace with a different mount namespace in order to debug.

I am aware, thank you :). I responded to a sibling dupe-comment over here [1].

To summarize, in my experience there is immense value to having basic shell tools available in the environment where you need them with zero extra friction. Stripping those out provides a security benefit only in specific nebulous and niche scenarios.

1: https://news.ycombinator.com/item?id=48561605

> in my experience there is immense value to having basic shell tools available in the environment where you need them with zero extra friction

I agree, however assuming you maintain a chroot for debugging this can be accomplished with a shell command that takes a single argument to target a running container by name.

Your linked comment suggests being limited to kubernates but nsenter and a chroot are entirely runtime agnostic.

You might not have any say on what image is in use, for example, in a cicd library project.

It’s handy when you’re troubleshooting issue on a running container which you can’t just rebuild the image and reload

Reply on news.ycombinator.com