Kubernetes and back – Why I don't run distributed systems
davd.io
davd.io
> I took a moment to calculate the cost of hosting my app on the K8s offerings of a Hyperscaler like AWS, Google or Azure just to figure out that this won’t be an option for me.
Clear signal that you can't justify the overheads. Cloud services are costly but in most cases you'll pay similar cost in terms of time and in some cases lot more if you pick something complex that you don't know about.
Even just something like EC2 instances is costly - and a surprising amount of stuff in the cloud is just containers or EC2 instances, before redundancy even. Adding in specialized services makes things even more costly.
Problem is that nowadays it's hard to find a team that knows how to properly handle hardware - if you manage to get a handful of people that know how to do that cloud is only sensible for very few edge cases.
And as the article states - in many cases you probably don't even need all that fancy from K8s. I can for the cost of using K8s cloud stuff just put a second copy of our hardware in a different DC (or maybe even a third one), and just fail between DCs if needed. Way simpler setup, and excludes a lot of failure cases I'd catch with a K8s setup.
From the homepage https://kubernetes.io/:
"Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications."
Do you see "not recommended for smaller-scale applications" anywhere? Including on the entire home page? Looking for "small", "big" and "large" also yields nothing.
if one wants to run service with high availability (no downtime if server died), what are other choices then?..
I've been in the SaaS application business since the early 2000's. When I first started, the big infra question was running on in-house infra or data-center infra. If it was the latter, then the follow-up question was which hosting provider. Regardless, as an Engineer, you were likely touching something that looked and felt like a variation of the box sitting under your desk.
Eventually we got cloud providers, serverless, containers, etc. Growing through the progression of technologies, they all seemed like a natural evolutiion of things to solve certain problems with other approaches.
For newer engineers starting out, they may only have seen these latter things, and at the scales and circumstances that make them a good choice. As such, it may not be that they're the new and shiny thing, but that they're the _only_ thing people have seen.
There's a difference between an EC2 instance and EKS pricing.
Bare minimum, one control plane, zero usage, no workload, EKS starts at $873 per year.
EC2 starts at $36/year. t4g.small (2 vCPU, 2GB) is $145/year.
A dedicated server from OVH running some processes under lxc and systemd can achieve 100% yearly uptime. Your hobby project isn’t Google.
73.6% of All Statistics Are Made up
Just go away and burn ;)
> A dedicated server from OVH running some processes under lxc and systemd can achieve 100% yearly uptime.
Sidenote: I used to take pride with these kinds of uptime. I now see it as a significant risk. If I take over a machine with an uptime of more than a month, I assume that the machine cannot survive a reboot. Which means I spend a lot of time trying to figure out how to deploy that server with all the data it needs. These days there’s likely deployment systems in place that prevent some of the horrors I’ve seen, but there was a time in the not to distant past when you had to assume someone ssh’d into a box at least once a quarter to muck with something. If they didn’t reboot, there was no guarantee the changes they made would survive.
I have just several servers here and there so their are all pets to me. Im 99% sure, server would boot cleanly. The remaining 1% is HW failure, with, can happen.
Example: I have a software FW running on some older HW here w/ 2 disks in Soft RAID 1. I had to power it down for 2 days due to electric maintenance in building. What a suprise was booting it. One disk failed right away making creepy noise. Other was ok, but server was booting long, so I hooked up monitor to check things out. Well, second disk was NOT in good condition either, it started to spit I/O errors here and there. Anyway. I managed to boot FW to working state, but immediatly got spare HW and started preparing for replacement. Now it runs 3 disks in Soft RAID 1 :)
Its not that simple. I care what I install on my servers. Everything is carefully selected. Additionally, I follow KISS concept, so I run simple things that are managable.
I have 2x 5$ VPSes on OVH for small side projects
>up 828 days
>up 1025 days
K8s is repetitive. You create one YAML for a service and the rest are similar. Rolling releases are a simple script. As small as our company and product is, I think K8s has been one of our best decisions.
Some things like MongoDB can quickly triple your costs though. We were able to pick technologies that kept the cost really low and stack maintainable.
Not the minikube version, not the kubespray version: the kubeadm version, installed by hand and minimally tweaked (NodePorts 0-65535), with dual-stack networking support.
I have ~3 tenants, 5-15 pods each, and managing this configuration with docker was a steadily increasing pain for the last few years:
- I had docker-compose split-files for each service for each tenant to keep a consistent state, so I used a simple bash script to generate a really long docker-compose line to bring everything together, networking was a real pain
- Dual-stack support in docker does not exist/work in compose v3 to this day, but v2 has it
- Querying the configuration and state of a container was an exercise in debugging the output
- For certificates, I had to run letsencrypt/certbot externally (2 tenants with separate accounts for wildcard certs), but use traefik in docker-compose (the syntax pain was on top of the rest, of course)
Kubernetes simplified all the pain points above:
- I still have the service split-files, but I don't care about network addresses and links between pods anymore, just their names
- better isolation between all the architecture points, and clearer definition and integration
- IPv4&IPv6 native just working
- "describe" gets me all the information I need in a pretty format, "get type name -o yaml" if I need the configuration
- standard nginx ingress + cert-manager, and I can have multiple tenants without resorting to hacks
DBs are regular containers in the pod that requires them, just like any other container. Daily subvolume snapshots are small and effective to make sure the data is safe.
Kubernetes is a much better orchestrator than docker compose, availability is a plus if you really need it. You don't need to build an HA control plane plus 3 workers if you have no use for it.
Many products have a minimum size at which they are effective, no one is buying an Oxide rack for their homelab or side project, no startup going through YC is buying an IBM mainframe, because they are both obviously nowhere near the minimum effective size/complexity/spend. K8s is the same, and yet it seems to trip up so many people.
For all your data: yes. It means persistent filesystems and databases.
Kubernetes will still be perfect to manage your running apps, but data is "the rest of the owl" of most devops / cd proponents and tools. "I should be able to rollback to a previous working state", well I hope your last deployment did not change your database schema.
MongoDB is shitsoftware anyway.
I use k8s in a very small setup (3 worker nodes, 3 ctrl planes) in my startup. But i'm working professionally with k8s for 6 years.
My setup is relative big but also relative simple and most of it is also IaC. I don't need to backup my setup and i don't need to document how i setup everything.
I have blue/green deployment, multiply enviorrnments (dev, test, prod) configured 100% equal, a good monitoring stack + tracing, s3 replication, postgresql ha and offside backup.
i can shut down a node and nothing happens.
But you know, if k8s doesn't fit your time and experience and benefit estimation, don't use it.
The upfront cost of k8s is bigger though.
Building my app, deploying a container image and building a helm chart is not a lot of work if you have experience and the initial additional work is only happening once.
After that, you have your whole app definition in code, seting up a second or third env takes 5 minutes.
I really should write a blog article about my stack and the advantges of it (git, kubernetes, argocd, helm, ...)
In addition to that Kubernetes isn’t really tailored to stateful workloads, so there are all sorts of gotchas and surprises there.
That's the right moral of the story to take away.
> Today, I’m back with a single machine that handles the peak 500 concurrent users
And this is what tells me k8s is probably not the right tool for the job. Something that runs on a single machine should absolutely run on a single machine... Unless if there's a pre-existing, funded and operated cluster you can tap into.
No abstraction comes for free, as the author mentioned you need to figure routing, backup, etc. You are paying a premium for the fact that you want to make that somewhat generic enough to fit with multiple use cases. The abstraction doesn't materialize for the operator of the cluster. It's there for the customers of that cluster - the devs who can now avoid to configure whatever the cluster is already handling.
With an operator, you get HA setup with switch over etc. basically for free.
I would prefer relativly cheap managed db over hosting it myself but its absolutly fine.