Kubernetes the Harder Way

github.com

24 pointscpach4 comments

Stumbled upon this yesterday and thought that people here might be interested. Seems very well-written and I was impressed by the quite radical bottom-up approach.

As the author of the guide, it's really nice to see feedback like this. Thank you!

Well thank you!

The funny thing is I was considering how to build a Kubernetes cluster using QEMU and a bottom-up approach. And since my main computer is running Apple Silicon I was ruminating regarding how to structure the intra-VM networking. I realized that it would not work the same as it would on a pure Linux setup. So I started googling like mad about vmnet-bridge & Co. Finding good info about those drivers isn’t exactly trivial, but eventually your guide appeared on a silver plate more or less. Neat!

VM network in QEMU on macOS works nicely. Unfortunately, the IO is not very performant, and that becomes apparent, e.g. when you run the k8s cluster with all the services that I described in the guide. The CPU usage becomes way too high for an essentially idle system.

The IO is much better with Apple's new "Virtualization Framework" (e.g. when using Lima with `vz` VM type - https://lima-vm.io/docs/config/vmtype/#vz), but unfortunately it does not support setting up a shared VM network - only bridged or single-VM NAT.

Reply on news.ycombinator.com