Hacker News Reader

Comment by catillac | Hacker News Reader

This is really interesting. Copying to the scratch container is powerful and I’ve used it a lot, regularly, but occasionally something comes up where I need to use a more fully featured base to support things. One other downside I’ve encountered with Scratch is no shell so doing docker exec or kubectl exec doesn’t work. Does anyone know a good solution to this problem?

View on HN

The Kubernetes folks' solution to this is the addition of `kubectl debug` (added as `kubectl alpha debug` in Kube 1.18, graduated to `kubectl debug` in Kube 1.20) as an alternative to `kubectl exec`. It takes an existing Pod and lets you attach a new container with whichever image you like, so that your production images don't need debugging tools.

Also, before `kubectl debug`[0] existed, you could always edit a Deployment and add a sidecar container of `alpine` or `busybox` and enable process namespace sharing[1] get some leverage to debug with.

A bunch of other options in the docs as well

[0]: https://kubernetes.io/docs/tasks/debug-application-cluster/d...

[1]: https://kubernetes.io/docs/tasks/configure-pod-container/sha...

That would trigger a restart, no? What if you want to live debug without restarting the scratch container?

Yup it would trigger a restart -- that first link details some other options if restarting the workload is absolutely not an option. You can debug a copy, start a privileged pod, or jump on to the node and actually enter the namespace manually. At the end of the day all these containers are just sandboxed+namespaced processes running on a machine somewhere, so if you can get on that machine with the appropriate permissions then you can get to the process and debug it.

Of course, if you're in a tightly managed environment and can't get on the nodes themselves things get harder, but probably not completely impossible :)

We drop a statically compiled BusyBox binary on images like that as "sh". If we need more we can symlink to it in /tmp at debug time (or just call it directly). It strikes a good balance between slim and debuggable.

Do you do this during build time or debug time?

Been using containers and fighting this problem for a long time... I never thought of this simple solution, thank you for sharing it!

In Kubernetes 1.22+, you can use ephemeral containers: https://kubernetes.io/docs/tasks/debug-application-cluster/d...

An emerging solution I've been investigating is using Nix as a build system for docker. The syntax is fairly lightweight and you can build containers that are scratch + coreutils, or whatever else you decide to put in there.

I was originally turned onto this by a post on the repl.it blog about using Nix this way.

You can also do this with other build systems or using weird docker file hacks by hand.

Here's some idea of what the syntax looks like:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-supp...

If you run a single process in a Docker container, have it output logs over STDOUT, and have no meaningful way to interact with it other than shutting it down, you don't have much need for a shell. It can't do anything in the container anyhow. By contrast, if you've basically got a full OS in there, then yeah, a shell is really useful.

I would say most of my Go projects involve what you described (logs to stdout, simple interface) but I often find myself needing a shell to debug my docker build, like ensuring that files have ended up in the correct place, and have the correct contents, within the docker container.

Use Dive for inspecting the image https://github.com/wagoodman/dive&ved=2ahUKEwips5OH0Lr1AhW97...

Your link is busted, https://github.com/wagoodman/dive

If your stack involves running containers on a Linux machine, you can use `nsenter` to use the debugging tools on the host OS to debug the processes within the container.

Distroless works well. Still no shell, but comes with “what’s usually missing in scratch”.

Instead of a container, just statically build the golang program with sqlite. Single binary deployment.

Default SQLite package requires gcc so you need to use one of the alternatives.

In the article they statically link in SQLite, It’s the reason the Go program can just be copied to a scratch image and still work.

You only need gcc during the build fase, that doesn’t change just because you run without Docker.

There’s nothing preventing you from statically link C programs either and have a single binary to deploy either. You just have to rebuild everything every single time security updates are a available. But that no different from Docker images, they need to be rebuild constantly as well.

Reply on news.ycombinator.com