Hacker News Reader

Comment by ssijak | Hacker News Reader

Is Kubernetes still hard in 2024?

Granted I'm new to devops, only been on a platform team for about 9 months now, but I still feel incredibly dumb every time I try to work with it.

That being said, we're also layering a bunch of stuff on top - helm, nginx, GKE, terraform, as well as a mountain of other things, and then to top it off we have a bunch of shell scripts doing random things to help tie it all together.

Normally I can pick things up pretty quickly. I just built a parser with tree-sitter, despite knowing virtually nothing about language design. Didn't take very long.

But the modern devops stack is a learning curve like I've never seen before. It's taking me more energy to learn it than it did for me to learn programming itself. Then again, maybe I'm just getting old.

I appreciate your shared experience as I have a rather similar one. I've been on a platform SRE team for about eight months myself, with seven years of SWE experience before that, and feel as though I'm just able keep my head above water. It's one thing to learn about k8s, the cloud, terraform, etc, then quite another to pile it all together, particularly since it all becomes heavily customized. It's a different job than code-writing software engineering, that's for sure. To me, it feels less like there's a 'stack' so much as there's a word cloud of DevOps buzzwords to start throwing at problems. Even when directed by architects & principals, it overwhelms.

I resonate with feeling incredibly dumb whenever I pick up a new ticket from our backlog. It feels like gaining deep knowledge of these systems will be a nearly insurmountable challenge. It's been eight months, and while I know far, far more than I did on day one, I feel that every day is a day one of sorts.

Depends (I know, I know...)

If you learn it from the top, immediately jumping through bunch of deployments, Helm (ugh) and other stuff?

You'll get fast to deployment but you won't know how to deal with things failing and there will be a lot of stuff that will remain "magic". Eady way to end up cargo culting despite best intentions.

Can be enough of you're just making simple apps to run on it but not running the clusteror the application in production. But you will have hard time understanding why things work and everything will be complex upon complex.

Go from the bottom up, learn basics of kube API patterns, kubelet, how scheduling works (don't have to be in depth), how kubelet works, how networking works (CNI, why kube-proxy is a bandaid for applications that handle networking badly, how services work), how storage works (how kubelet mounts things to containers, etc). Then how higher level controllers (aka operators) work - from Pods, through ReplicaSet to Deployment, StatefulSet, DaemonSet.

This way you'll learn the basic building blocks, which are quite simple despite the long list I just gave, because the architectural and API patterns repeat and build over each other. The core is simple which lets you build complex stuff on top while still understanding it.

From recent experience I'd say it's the sort of tech that starts off simple enough with the right distribution, but then gets more complicated the deeper down you dive.

Probably the most hard thing I found was wrapping my head around the way the storage works.

I can only speak from the perspective of somebody trying to manipulate k8s environments for use in test, but I'd say yes.

Four times across three companies I have run across frameworks which set everything up out of band and then ask the tests to test it. Maybe these are shell scripts written by the k8siest person on the team. Maybe they're something like tilt. Whatever the case they're always a black box to the majority of the people who are writing application code.

They get you 90% there, but eventually somebody wants assurances that some environment variable has the desired effect, and suddenly you need to penetrate that black box and change it so that there are multiple kinds of "up" and the right tests run against each state.

K8s tooling is commonly installed via curl, so once you unravel the black box and integrate it with your tests you end up with a lot of fragile interfaces to things like kustomize, kubectl, kind... Fragile because maybe the other dev has a different version installed. Nix dev shells solve this, but you can't usually get the whole team on board with Nix so version mismatches come up often and are often difficult to debug. You end up in a state where whoever wrote the initial setup scripts is authoritative about the dependencies, and you have to ask them what they have installed if you want things to work (it was easy for them, they just used whatever was lying around at the time).

These aren't directly deficiencies of k8s, once you see the light (which takes a long time) it's pretty easy to work with, but like so many other technologies, the devil is in the peripheral tooling and the culture. K8s doesn't (yet?) have a very nice boundary with other language ecosystems, it reminds me of Java in that way. The die hard k8s people often want to solve problems by bringing them more fully into the k8s way of seeing the world and I just don't think that is consistent enough with reality to be our everything.

I've cultivated a begrudging respect for it, but I still don't like it. If I break free and start my own company, I'll publish an operator so that my stuff can be installed into k8s, but I don't intend to make it primary in any way.

I've seen projects run on Cloud Run because they fit that model. But if you don't quite fit the model, then you fight the abstraction in odd ways. K8s gives you control over more things, so the easy stuff feels hard but the harder things aren't as hard. With k8s operators you can stand up pretty complex things very quickly and robustly.

IME yes, yes it is. I usually tell people running a k8s cluster is similar to running your own "cloud". If you want to just deploy an EC2 instance you just tell AWS you want an EC2 instance and you are done (mostly). You don't have to worry about if the hardware under the EC2 VM has enough resources, you do have to worry about that with k8s though. If you want to lock down the EC2 VM to have certain permissions you use AWS IAM, with k8s you have to use cluster roles and cluster role bindings. You can apply these 2 examples to many other things in k8s vs "cloud" provider, Ingress, persistent volumes, etc.

I heard people say its not for small teams. On top of that I'm not convinced it's a good tool for anything "storage" (i.o.w. use it for compute/network loads).

It really depends on the design of the workload. Kubernetes makes orchestration more simple, but it doesn't remove all of the complexity of orchestrating something.

If you don't need much orchestration (which is true for a lot of postgres users), the complexity from kubernetes is compounded on top of the complexity from postgres without generating much value.

I don't know about hard but it's fairly straightforward to spin up clusters and maintenance seems to have become less of a headache (at least imhe). It depends on what you will be doing with the cluster and how you use it.

Yes. Orchestration is not easy.

Reply on news.ycombinator.com