Hacker News Reader

Comment by koeng | Hacker News Reader

Anyone know how this compares to k3s? I’ve been using k3s for a while, and there have been a few bugs that made me a bit annoyed

View on HN

This doesn't answer your question, but I'm piggy-backing with comments on k3s.

For me, using k3s for development (not prod), the killer feature is running it in --docker mode where the node uses the local dockerd to run containers (vs. managing its own containerd instance).

This allows building images locally with `docker build` and immediately using them in kubernetes pods _without_ first pushing to a (possibly local) image repository and having k3s pull the images.

Last time I investigated, none of kind, microk8s, or minikube supported this mode. For large images (gigabyte or more, and I've got a handful of these), it's very space-inefficient to have a copy in my docker and in a local registry _and_ in k3s's containerd at the same time. How is this problem typically solved?

(I note the kubernetes included in Docker Desktop on macOS works in the same way: images built with `docker build` are available to kubernetes without going through a registry.)

We're using img [1] with a helper script. The helper script runs a pod in the (potentially) remote Kubernetes cluster that takes the container build context as stdin, then runs img to build the image, and then imports it into the host's containerd instance. It also maps in the right host volumes so that the image build is cached between subsequent runs.

I was surprised to be unable to otherwise find a good local / remote container development workflow, but this was built to replace what we were previously doing, which is setting DOCKER_HOST to point to the remote (single-node) cluster's Docker daemon (over SSH), so that docker CLI commands would execute on that remote box.

In both cases, you'll still want to take steps to minimize the size of your container image build context, but the size of the images doesn't matter. I'm not sure if it'd fit your needs or not, though.

[1] https://github.com/genuinetools/img

FYI minikube supports this via minikube docker-env

https://minikube.sigs.k8s.io/docs/handbook/pushing/#1-pushin...

A couple of years ago minikube supported this with --vm-driver=none

Still does: https://minikube.sigs.k8s.io/docs/drivers/

it's a bit more troublesome with microk8s https://microk8s.io/docs/registry-images

[I work on kind amongst other things...]

If you want to turn your host into a node you can do this with --vm-driver=none in minikube or better yet just use `kubeadm init` directly. In KIND we point people to the latter -- the main thing we're doing is running inside a disposable container "node" of which you can have many.

Assuming you don't actually want to turn your host machine into a node managed by Kubernetes, you'll want to stick Kubernetes in a VM or container. If the rest of Kubernetes is in this container or VM, it doesn't make sense to be running containers out on the host, things like mounting volumes won't work, you need a consistent filesystem between kubelet and the container runtime.

With kind it's also important that we simulate multi-node and multi-cluster, which is not viable with a single container runtime instance.

Without actually running Kubernetes against the hosts's runtime you can't share storage. The way docker desktop does this is to run Kubernetes with docker as the node's container runtime while only supporting a single node/cluster in a VM and expose the same runtime for building.

For our test workloads it's important to have different clean clusters constantly for different tests / projects.

KIND and microk8s have made a bet on containerd, as kubernetes is actively moving away from dockershim towards CRI, so even if we exposed a node runtime you can't build with it.

It's indeed space-inefficient, but it's a tradeoff in isolation between projects etc. For multi-node you're going to wind up with multiple copies anyhow, and a lot of projects we work with wind up needing some multi-node testing.

The first thing I think of with microk8s is their available addons, which are fairly comprehensive, opinionated, "just works" solutions for microk8s. https://microk8s.io/docs/addons#heading--list

As a k3s user (for local dev & my very small personal prod envs), I end up having to assemble a lot of these solutions myself. I prefer my own picks, my own solutions, and learning, but microk8s having these instantly available would be really good for a lot of folks. Until now though it has never felt like those advantages would be useful in a real prod environment, that microk8s was not interested in being in prod, but HA signals to me that they are interested in broader adoption.

I'm running k3s in production. K3s has hooks to setup Prometheus, autoscaling (for spot instances), etc.

I don't see all of these in microk8s. I'm not sure if this is on the roadmap.

I'm also not sure how customisable microk8s is. We run k3s with haproxy ingress (which is not the default) and calico for network (again nog the default)

sadly you need to run with kube-proxy and can't use calico's own sauce. but I already raised an issue for that, if I would use it more instead of kubespray (which I will deprecate soon) than I maybe fix it on my own time.

microk8s has a Prometheus add-on that you can enable with one command: https://microk8s.io/docs/addons

I'm using it and it's been great so far.

Several commenters here say they have fewer bugs with MicroK8s, makes sense since it’s less of a modification than k3s. Slightly bigger since it keeps api and worker processes as separate binaries albeit in a single package.

It uses distributed sqlite (dqlite) vs etcd/RDBMS (postgresql and friends). It's also not part of the CNCF like k3s is (yet?).

dqlite is a RDBMS, it's just sqlite + raft

k3s swaps etcd for kine, which is an app that implements enough of the etcd API to support kubernetes. It uses dqlite under the hood, but you can use a different database if you want (though it's not clear to me if you still get Raft if you do so).

It looks like Canonical is just using dqlite directly.

Same question, but with minikube.

I think minikube is for local usage. I use K3S for deploying an actual production cluster, so I’m more curious about how micro-k8s acts on that scale

> I use K3S for deploying an actual production cluster

You mean when you want to run a small cluster of let's say less than 10 nodes (anything in single digits)?

Why doesn't normal k8 work this way? like same tech but just less scale?

(I should probably read more on the side myself as well, new to this K8 world).

Yep, it’s a small self hosted cluster on my own server hardware (VMs on Proxmox) I’m sure k8s would work, but k3s made it really easy for me to get going so now just has inertia.

I see, I assumed logically Kubernetes as a community would want to keep thing simple and keep it same across scale. For large scale there could be different variations (like switching storage backend) but for small/low scale tier keep it dead simple.

I am guessing there were valid reasons for the offshoots (k3s, microk8s etc.).

It does. Upstream kubeadm - authentic kubernetes - can even run on a single node. Not sure why people choose k3s or microk8s when you can just as easily deploy the real thing.

For smaller needs, K3s runs fast/stable on servers with 1-2 GB of RAM, whereas K8s proper tends to be a little shaky until you go to 2-4 GB, minimum.

I hear this a lot but have any actual, tangible comparisons been done with k3s vs kubeadm arm resource use?

kubeadm is very bare bones though; it gives you a running node but you are still responsible for configuring network and storage providers, an ingress controller and probably a load balancer.

That's exactly what i am confused about when only reading these things in isolation.

Reply on news.ycombinator.com