Canonical introduces high-availability Micro-Kubernetes

In general, I'm not blown away with Canonical's track record. I used microk8s in its early days and it kind of blew up my coworker's computer. Networking stopped working. Making a request to localhost:5000 would return an nginx error page, even though nginx wasn't running anywhere on that machine or the network. It even persisted itself through reboots! We just reinstalled the machine eventually. It was weird stuff and I haven't touched it again. (I prefer kind locally and k3s for small "real" clusters.) Then there's Snap, major breaking changes to Ubuntu for no reason, etc. I just don't trust Canonical much, and I'm not quite ready to take a leap of faith on their new distributed database. But maybe it's great, and we'll all be using this in a year. A stopped clock is right twice a day.

The only place etcd might not pay off is in disposable dev environments or something, but do you really want your prod setup to page only to discover a complete cluster rebuild is necessary to resolve the problem?

though with kine (kine is not etcd) you can easily use mysql/sqlite/dqlite/postgresql.

Surprisingly, Etcd is neither scalable or high performance in our small cluster with <10 nodes on GKE. It had been the single most frequent source of prod outages.

But it appears that k8s community do not section a list of compatible storage engine that can plug and play with k8s. Or I might have missed some recent development.

The only remaining headache would be database. Setting up reliable mission-critical HA Postgres is kind of a pain, and it's nice to not have to worry about it. If that can also be canned, then this kind of rig would be a really compelling alternative to the managed clouds... unless you need other things like S3, etc.

i tried both in my homelab and this is my experience as well. microk8s seems to has less bug for me.

As a k3s user (for local dev & my very small personal prod envs), I end up having to assemble a lot of these solutions myself. I prefer my own picks, my own solutions, and learning, but microk8s having these instantly available would be really good for a lot of folks. Until now though it has never felt like those advantages would be useful in a real prod environment, that microk8s was not interested in being in prod, but HA signals to me that they are interested in broader adoption.

For me, using k3s for development (not prod), the killer feature is running it in --docker mode where the node uses the local dockerd to run containers (vs. managing its own containerd instance).

This allows building images locally with `docker build` and immediately using them in kubernetes pods _without_ first pushing to a (possibly local) image repository and having k3s pull the images.

Last time I investigated, none of kind, microk8s, or minikube supported this mode. For large images (gigabyte or more, and I've got a handful of these), it's very space-inefficient to have a copy in my docker and in a local registry _and_ in k3s's containerd at the same time. How is this problem typically solved?

(I note the kubernetes included in Docker Desktop on macOS works in the same way: images built with `docker build` are available to kubernetes without going through a registry.)

It looks like Canonical is just using dqlite directly.

I don't see all of these in microk8s. I'm not sure if this is on the roadmap.

I'm also not sure how customisable microk8s is. We run k3s with haproxy ingress (which is not the default) and calico for network (again nog the default)

It doesn't use a ton, but in order of least-to-most lightweight IME it's Minikube -> Microk8s -> k3s

But again, the overhead is marginal so it's not a world of difference.

If you can run Swarm you can definitely run one of the lightweight k8s distributions.

Of course it does shift the complexity into the underlying K8S so installing and operating it can be difficult but there are manys to avoid that as a user. Overall you gain much more in productivity and usability, which is why it's taken off so much.

Gotta say, the docker ps output on those machines looks like line noise to me.

Not all movement is progress. Young developers are incentivized to embrace new technology because it levels a playing field where time with a tool is your most important asset. That playing field is not real, but a lot of managers seem to think it is, so the strategy works. I don't know how we sell a different version of reality there, but we need to figure it out.

At my first big job, the oldest developer told me shortly before he left that essentially we keep facing the same set of problems in a loop, and that if I watch for it I'll see it happening. That was in many ways a very big shoulder to stand on.

That challenge, added to historical information I had learned in a class on distributed computing, changed my perception of my first loop, like I'd found a shortcut. In my second loop, I found myself having productive conversations with people on their third loop, while my coworkers were still chirping on about how it's going to be different this time.

We just keep playing a game where the rules (like cost inequalities between resource classes) get tweaked every game, but quite often they revert back to the previous rules in the following game (because the people who make hardware for that resource finally figure out how to fix their bottleneck). But instead of recycling or democratizing the tech that worked last time the rules looked that way, we reinvent it badly with new names.

Elixir is a rare exception in this case, which is part of what attracts me to it. It's essentially recycling 25% of Erlang and 45% of Rails and being transparent about it, creating a new recipe out of old ingredients that have worked well in the previous 4 tech cycles.

That is why we are stuck in fashion industry nowadays.