I don't think I want to use kubernetes (or anything that uses it) again. Nightmare of broken glass. Back in the day Docker Compose gave me 95% of what I wanted and the complexity was basically one file with few surprises.
K8s isn't for running containers, it's for implementing complex distributed systems: tenancy/isolation and dynamic scaling and no-downtime service models.
- Kamal
- Docker compose with Caddy (lb_try_duration to hold requests while the HTTP container restarts)
- Systemd using socket activation (same as Docker compose, it holds HTTP connections while the HTTP service restarts)
So you don't have to buy the whole pig and butcher it to eat bacon.
Nit: it holds the TCP connections while the HTTP service restarts. Any HTTP-level stuff would need to be restarted by the client. But that’s true of every “zero downtime” system I’m aware of.
It's far from table stakes and you can absolutely overengineer your product into the ground by chasing it.
"0 downtime" system << antifragile systems with low MttR.
Something can always break even if your system is "perfect". Utilities, local disasters, cloud dependencies.
I'm still using it with not a single issue (except when is messes up the iptables rules)
I still confidently, upgrade the docker across all the nodes, workers and managers and it just works. Not a single time that it caused an issue.
I heard good things about Nomad (albeit from before Hashicorp changed their licenses): https://developer.hashicorp.com/nomad
I got the impression it was like a smaller, more opinionated k8s. Like a mix between Docker Swarm and k8s.
It's rare that I see it mentioned though, so I'm not sure how big the community is.
Everything else is composable from the rest of the hashicorp stack consul(service mesh and discovery),vault(secrets) allowing you to use as much/or as little as you need and truly able to scale to a large deployment as needed.
In the plus column , picking up its config/admin is intuitive in a way that helm/k8s never really comes across.
Philosophy wise can put it int the unix way of doing things - it does one thing well and gets out of your way , and you add to it as you need/want. Whereas k8s/heml etc have one way or the high way - leaving you fighting the deployment half the time.
It's a shame Nomad couldn't overcome the K8s hype-wagon, but either way IBM is destroying everything good about Hashicorp's products and I would proceed with extreme caution deploying any of their stuff net-new right now...
Using it in prod and also for my personal homelab needs - works pretty well!
At the scale you see over here (load typically served on single digit instances and pretty much never needing autoscaling), you really don't need Kubernetes unless you have operational benefits from it. The whole country having less than 2 million people also helps quite a bit.