Domesticating Kubernetes
blog.quickbird.uk
blog.quickbird.uk
Kubernetes drags in a tremendous amount of complexity, background knowledge, and strange constraints, even on small installations. I really do not see its benefit for small apps and projects.
Are you hosting your blog? You don't need k8s. Are you running a small app server at home? Nope, don't need it. Are you running an auto-scaling app with many hundreds or thousands of worker instances, and need to support non-disruptive roll-in upgrades across multiple clouds and have many services which need to scale independently? Maybe you need k8s, but only in some circumstances.
For the record, I run many k8s clusters across several clouds, using 1,000 - 50,000 cores at any one time, so I've dealt with quite a bit of k8s complexity, and I'm still on the fence whether it's the right answer for us, but it's allowed us to standardize our software to k8s and only worry about getting that running well in each deployment, which put the cross-cloud work on the infrastructure teams versus the software development teams. The price you pay for this is that you still need to do non-k8s stuff if you need some cloud specific resources more than simple compute and routing, so you have both k8s and cloud specific code.
Can you have Docker do that too?
One thing I found in running my home lab was that I kept having to burn it to the ground & rebuild it regularly anyway. For example, dist-upgrade never really works: every time I try I just end up wasting a couple days wrestling with it and then giving up and rebuilding the machine from scratch. Even if I just assumed that I'd have to build from scratch every time, differing app and library versions meant that I couldn't count on a new build being a simple, clean install.
So going with the regular "run things as a daemon on a server" model wasn't actually saving me that much time.
Basically I could to do one of two things:
1. keep using purpose-configured machines and spend a bunch of time writing ansible scripts to automatically re-create them when everything goes pear-shaped and then re-write the scripts when a new version changes stuff.
2. container-ize all my tasks, and make everything else vanilla and effectively disposable. I have to spend some initial time to rewrite my stuff in container-speak, but that's a one-time cost.
Presented like that, option 2 looked like a better option. When a machine has a problem or needs to be rebuilt, I build it with the completely vanilla setup (ubuntu lts, conjure-up k8s) and push my k8s jobs and pods up to it. That's 2 hours instead of a day and a half. (Yes, in theory I could docker-ize everything and run docker-swarm, but it's a small step from there to k8s, and conjure-up makes installing k8s fairly straightforward.)
Frankly, I have a family now, and fucking with fiddly settings and library dependencies isn't fun anymore. I'd rather spend that time doing stuff. k8s lets me divorce my hobby work from the infrastructure, and I like that.
k8s is probably great for many things, but as time goes on I really appreciate the ability to debug things effectively and by keeping my homelab setup as simple as possible I avoid the complexity of whatever advanced solution that is out there.
Not understanding how k8s work, or actual provider screwing up?
if the former - I try to read carefully and test my workloads in a test cluster before I do it in prod.
if the latter, I think google has excellent operational excellence.
This is like saying you wouldn't take driving advice from me until I total a few cars. Perhaps I didn't total a few cars because I make good driving decisions?
>>This is like saying you wouldn't take driving advice from me until I total a few cars
No, I am saying you do not have enough experience with k8s. That is all. Totaling a car implying you are driving, using k8s implying that whoever wrote that particular part of k8s is driving. You have some control but ultimately you need to understand what is going on. As that list shows many companies who put k8s into production learned it in a hard way.
Should I compile a list of failure stores for companies that ran Linux because Linux is complicated or has had various bugs?
How about a list of failure stories of how Ruby causes problems?
You can get errors with any system. You would need to prove to me that k8s has say more errors per day than a competitor (lambda? VMs). I just don’t understand...
This is the claim that you made. I would argue that since you do not have experience with it you perceive it as you do not need to focus on your infra. Once you run into a failure with k8s and you actually need to fix it, this might change. That is all.
I have run into Linux failures. I still use Linux.
I have ran into bugs in Java. I still use Java when it makes sense.
So not sure what you are trying to get to. You give up on a technology the first time you have a problem?
If I want to shave pennies, I have many places to shave before the very thing that is the heart of my business.
The only reason I’ve found to run k8s at home is if your full time job is operating k8s at work.
Also:k3s might be an option.
If you run your home server on a separate box, like a RaspPi or NUC, docker-machine via SSH can make provisioning and updating a tad easier.
Were I learning k8s for work, it wouldn't be a good solution, but for simple home apps that you want to fire and forget, it definitely beats doing a hand-made installation and periodically care-and-feeding them.
Only reason I can see RPIs in kubernetes is if you're exclusively using ARM everywhere and/or are running some distributed cluster among different locations (like Chick-fil-a).
I recently did this at work, using a single Dockerfile to built multi-arch images for ARM and x64 - it's pretty nice!
I used RPis for my robot (https://sendc.at/dl/Kjjbt3dij6T733YgWsyv3p3Vv5GPGKO1q5IEFStV...) with daughter boards for motor control) and they wre pretty convenient, but never really considered running anything server-like on them because they have reportedly woeful network performance.
It's pretty powerful for the cost and has no fan etc. Arm64.
Then you could type (for Plex, for example) plex.k8s.lan
Maybe we need the equivalent of traefik for DNS?
Aside from that, I don't think what he proposes is a great solution, because what we'd need is the automated way for deployments to get that DNS created (or announced) for their IP when they get it or when it changes. Having it done manually and being static is vastly different in usability from k8s does with ingress/cloud controllers.
It's the first service that goes up after you've initialized the cluster and initially serves only internal requests.
Nothing stops you from pointing the matching lookup zone to the internal dns of k8s, however. I've done it before and it worked great for lan requests.
If you want to expose it to the internet however, an automatically configured dns is probably not what you want, unless you actually have a public ip range to use with said services. In that case, the original comment makes more sense and you'd just add a wildcard dns to your ingress controller, which can be traefic or whatever else you want
- specify a LAN IP for your ingress controller so it doesn't change.
- Use ddwrt/dnsmasq to point *.k8s.myhomenetwork.local to said IP
Once that's configured, you just configure the ingress hostname on services as you would "normally".
> This document specifies that the DNS top-level domain ".local." is a special domain with special semantics, namely that any fully qualified name ending in ".local." is link-local, and names within this domain are meaningful only on the link where they originate. This is analogous to IPv4 addresses in the 169.254/16 prefix or IPv6 addresses in the FE80::/10 prefix, which are link-local and meaningful only on the link where they originate.
Microsoft used to recommend .local TLD for AD deployment as a best practice, and nowadays there are companies stuck with this decision. Do not make the same mistake; unlike companies, you probably want your zeroconf stuff to work.
How it will behave will depend on your specific stack. Zeroconf aware (Macs, iOS devices, Linux with Avahi - i.e. most modern distributions) one will use multicast, zeroconf unaware (Windows) will use your DNS resolver. Devices (printers, etc) are a toss of a coin.
I then have external-dns running (https://github.com/kubernetes-sigs/external-dns) which manages the relevant A/CNAME records on Google DNS (other DNS providers are supported) so that I can resolve "myservice.mydomain.com" to the service's IP address.
I wrote a bit about the BGP bit last year: https://www.growse.com/2019/04/13/at-home-with-kubernetes-me...
Admittedly, I have no desire to expose any of these service to the internet, but if I did I could use an IPv6 address on the service instead, or add a static NAT rule to the router to forward traffic to the service IPv4 address. Auto provisioning of NAT rules feels icky, so I'd probably go down the ipv6 route if I wanted to do this.
I have a static LAN IP for an Ingress Controller, say 192.168.2.100. All HTTP/HTTPS requests are port-forwarded to it from my router. From there, the Ingress is in charge of directing the request based on domain name.
As for DNS, I use a single domain name, and have a record for a wildcard subdomain - so any subdomain will end up at my router, I don't have to configure anything at my DNS registrar when I add yet another application, as long as it's using a subdomain.
ExternalDNS is a superior solution, but most people will only have 1 or 2 domain names.
If it's experience deploying applications into containerized environments, then micro-k8s and k3s seem like reasonable choices, you don't really care about the setup of the underlying components, just that they present the k8s API.
If you're looking for experience of managing k8s clusters, then either the distribution you're looking to run in prod. or something like kubeadm are perhaps a better option. kubeadm is very "vanilla" in terms of how it's deployed so it's quite representative of production (on-prem) deployments, perhaps unlike k3s which makes changes to how k8s works.
If you're looking to quickly test things in k8s, I'd recommend kind as the easiest way to stand up and remove clusters quickly.
And if you're looking for something to run your home services long term, I would recommend not using Kubernetes :) (unless you have a really complex home network which might justify adding k8s to the mix)
If we disregard one’s experience with Kubernetes as a factor, are there any other reasons you see to not use k8s at home?
So outside of playing with Kubernetes for experience, why would you do that?
From a complexity perspective, you're adding more places for things to break. Instead of (say) running your apps in containers with Docker, you add pods, services and ingress as layers, which is more places for things to go wrong.
* "I want to run my stuff from home indefinitely, in the background": Don't use kubernetes. You'll spend ages setting up your cluster, you'll be fighting to keep it up. Most things you'll want to run probably have an OS package and run happily as systemd services. It's unnecessary overhead, both on the hardware, and mentally. Internal certificates [used to, at least] expire after a year on kubeadm created clusters, so if you don't look at it occasionally your control plane goes down. It doesn't handle the case where all your machines are nearly at full capacity (i.e. >85% disk, or nearly all memory used) and the defaults are to kick things off nodes with "pressure" which is a huge PITA when you don't have a dynamically scalable cluster - nothing makes my blood boil than seeing 200 lines of "Evicted" in kubectl . It's designed for huge workloads in datacentres, after all. You really don't need it for hosting a single user blog and a NAS.
* "I want to set up a homelab and learn k8s": Definitely use kubernetes. You'll learn how painful and time-consuming it is to manage onprem installs, but you'll also learn a lot. A lot of packaged kubernetes solutions follow the same patterns (ingress controllers like nginx / ambassador / envoy, istio service mesh, flannel etc for VLANs, prometheus for monitoring, helm / argocd deployments, ...) so it's super useful if you need to use kubernetes at work. You'll come to realize just how much awful, awful stupid rolling-release "this was deprecated on Tuesday and all our config schemas have changed" bullshit you're protected from when using someone else's (i.e. BigCorp's cloud solution) managed kubernetes cloud thing, and screaming "NO" at anyone who starts to utter "on-prem" will become ingrained in you.
:-)
Regarding Kubeadm - some storage solutions don't even work on K3S, or at least i don't see how you can make them work. For example EdgeFS Rook integrated CSI driver requires you to deal with feature gates.
Docker compose is really great but only works on a single machine. Also docker compose makes rolling updates a PITA.
What I really want is something like Cloud Run but locally tunable.
Google Cloud Run is essentially: here is a docker image, when you get a http request, spin up a container, when there are no requests for 10 minutes, spin it down. If there are more than 80 req/s, replicate to handle the spike.
Now I want to say, here’s a bunch of Ubuntu machines with X cores, Y RAM and Y SSD storage. Go run these images on it and auto scale it up and down.
I know k8s was meant to solve this problem but the layers of abstraction on top of abstractions are insane.
I just want to specify my compute/storage pool of machines, my docker images, how they should connect to each other and scale up/down. Boom! It just works.
Is there something that does this?
https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/
here's a live walkthrough - https://www.youtube.com/watch?v=DjpVtNjiXSU
in what way?
Last I checked, K3s is a fully certified distro of k8s.
I don't want to make this into an anti stance, but we have totally loved the experience of using k3s.
The way you use k3s generally is not like the way you read on this post. It's basically a single command experience with sane defaults (e.g traefik). The overall experience is very close to a single click experience...all the way to the cloud.
Also, K3S is not just faster - storage solutions don't even work on K3S, or at least i don't see how you can make them work. If you try to install EdgeFS, it's integrated CSI driver requires you to deal with feature gates.
My setup did work initially. I had a dedicated server acting as the metal lb, about 20$ a month. That was then connected through a wireguard tunnel to my home dmz network. Which backed to a dual xeon v2 work station. The latency was very good under 20ms, and really good speeds. I'm lucky to have fios in my area.
The Xeon workstation died, I then backed up to several think pads. Those were not performant either. So I got several HP t610 nodes, raspberry pi speed with sata 3. Rook took all CPU, and to boot they were running at 90c constantly. Even after re paste and with fan mods. I didn't want five little space heaters next to my desk.
After all this I ditched the home setup. I had gotten parts for a local Epyc server after my xeon's died. But sold them due the current situation.
In the end I had wanted to start a series of blogs on Kubernetes and micro service development. To help me learn and flesh out my under standing of Kubernetes.
I don't feel I wasted several months setting up Kube. I now know about under the cover stuff. Having deployed OkD, Rancher, kubespray, and kube adm. The initial wire guard set up helped cement a lot of the internal networking model. I was mildly acquainted with having worked on openvswitch and open stack before.
If you're doing local. I would really recommend a Ryzen 1600af (6c/12t zen+ 65w), either asrock rack x470, or a basic b450. Both can take ECC, and should land a little node under 500ish or so. There are also SFF PCs Lenovo ThinkCentre M93, comes to mind. But at that price 90$ a node, I'd rather move up the stack a little bit.
If you're waiting to buy local. DigitalOcean has been very well priced. If you want to learn the internals, grab a dedicated server and set up KVM, to get familiar with the internals. On the upper end you can get a new ryzen 3rd gen dedicated for 90$ a month. I look at it as a 90$ class I take once.
I'm not saying Kube is always right. But I avoided it for a long time, backing up to docker swarm. Now, I feel at a base it's not much more complicated than docker swarm, now that I've done this deep dive. Kube adm is on ease of use with docker swarm to me. Add in Metal LB, and Gitlab to help manage the cluster, you have a personal little cloud. It's also good to know for future job searches.
I don't think it beats anything, but I am sure IO improves fairly significantly.
One of the things I found to be a problem is that most container images found on different registries are built for x86_64. You would need to rebuild those containers yourself on the RPI.
i've been using k8s at work for over 4 years but for home usage, it's just too much.