Comment by pmahoney | Hacker News Reader

Comment by pmahoney | Hacker News Reader

Parent Full thread

This doesn't answer your question, but I'm piggy-backing with comments on k3s.

For me, using k3s for development (not prod), the killer feature is running it in --docker mode where the node uses the local dockerd to run containers (vs. managing its own containerd instance).

This allows building images locally with `docker build` and immediately using them in kubernetes pods _without_ first pushing to a (possibly local) image repository and having k3s pull the images.

Last time I investigated, none of kind, microk8s, or minikube supported this mode. For large images (gigabyte or more, and I've got a handful of these), it's very space-inefficient to have a copy in my docker and in a local registry _and_ in k3s's containerd at the same time. How is this problem typically solved?

(I note the kubernetes included in Docker Desktop on macOS works in the same way: images built with `docker build` are available to kubernetes without going through a registry.)

[I work on kind amongst other things...]

If you want to turn your host into a node you can do this with --vm-driver=none in minikube or better yet just use `kubeadm init` directly. In KIND we point people to the latter -- the main thing we're doing is running inside a disposable container "node" of which you can have many.

Assuming you don't actually want to turn your host machine into a node managed by Kubernetes, you'll want to stick Kubernetes in a VM or container. If the rest of Kubernetes is in this container or VM, it doesn't make sense to be running containers out on the host, things like mounting volumes won't work, you need a consistent filesystem between kubelet and the container runtime.

With kind it's also important that we simulate multi-node and multi-cluster, which is not viable with a single container runtime instance.

Without actually running Kubernetes against the hosts's runtime you can't share storage. The way docker desktop does this is to run Kubernetes with docker as the node's container runtime while only supporting a single node/cluster in a VM and expose the same runtime for building.

For our test workloads it's important to have different clean clusters constantly for different tests / projects.

KIND and microk8s have made a bet on containerd, as kubernetes is actively moving away from dockershim towards CRI, so even if we exposed a node runtime you can't build with it.

It's indeed space-inefficient, but it's a tradeoff in isolation between projects etc. For multi-node you're going to wind up with multiple copies anyhow, and a lot of projects we work with wind up needing some multi-node testing.

FYI minikube supports this via minikube docker-env

https://minikube.sigs.k8s.io/docs/handbook/pushing/#1-pushin...

We're using img [1] with a helper script. The helper script runs a pod in the (potentially) remote Kubernetes cluster that takes the container build context as stdin, then runs img to build the image, and then imports it into the host's containerd instance. It also maps in the right host volumes so that the image build is cached between subsequent runs.

I was surprised to be unable to otherwise find a good local / remote container development workflow, but this was built to replace what we were previously doing, which is setting DOCKER_HOST to point to the remote (single-node) cluster's Docker daemon (over SSH), so that docker CLI commands would execute on that remote box.

In both cases, you'll still want to take steps to minimize the size of your container image build context, but the size of the images doesn't matter. I'm not sure if it'd fit your needs or not, though.

[1] https://github.com/genuinetools/img

it's a bit more troublesome with microk8s https://microk8s.io/docs/registry-images

A couple of years ago minikube supported this with --vm-driver=none

Still does: https://minikube.sigs.k8s.io/docs/drivers/

Reply on news.ycombinator.com