Hacker News Reader

Comment by powerhour | Hacker News Reader

This is more a criticism of operators than StatefulSets. The promise of operators is easier automated management of your resources. If the operator can't do basic things like resizing disks, it's flawed and needs to be fixed.

However, I've always been suspicious of operators, so maybe my bias is leaking here. Kubernetes is already a level of abstraction and adding another seems risky and unnecessary.

View on HN

And now imagine it’s all in a Helm chart. And using kustomize magic

So you start debugging the helm chart, and kustomize, and the operator.

Not fun.

Your service is now crippled by a bunch of spaghetti APIs on top of the stuff that was crippling it before. You are cloud-scale!

Now that you've hit cloud scale, engineering leadership can make the case for budget for an ops hire to keep the thing working. Bigger headcount is always a plus.

Also make sure that Dev and Ops are separate departments that are not allowed to work together. Devops? We don't do that here. BTW you need to fix that 15-second query ASAP.

> Bigger headcount is always a plus.

You can't make a baby in 1 month with 9.5 women.

Listening to the empire builders among engineering management, they'll happily tell you that a few more hires will get the current baby back on schedule plus future babies can be done in 8, maybe even 7, months.

I agree with this. I manage a monorepo of 10s of terraform/terragrunt modules most of which are deployed to a k8s cluster.

the whole point of code as infrastructure is that your git repo is an audit log of whats changed in your infra. Once you leave it up to operators or persistent volume claim templates (or whatever else makes side effectful changes with being told) you're throwing all that niceness of control out the window.

A Git repo can still be a good audit log even if you use templates, especially if you're adding a step to transform files with each push.

Infra as code can't actively manage things like database clusters, handling promotion of a secondary, automatic deployment of new replicas, scaling. Using it to deploy RDS Aurora doesn't count. My impression is that operators are designed to handle those "active" decisions in real-time, and that's outside the realm of terraform, GitOps etc.

Indeed, and if the k8s api was so precisely laid out that you could look at a log of side effect changes that came as the result of scalers, operator, templates etc that would be one thing, but we aren't there yet.

Right now what i have is "oh the cluster is in a different state (there is drift) than the code, lets be a detective to see why this changed".

To be fair there are tools that use operators (like Flux) to apply desired state onto Kubernetes clusters using Git, so the GitOps is still there. You can even combine flux with ClusterAPI to create clusters with CRDs through Git, which can be quite handy

As long as the operator is really well designed but it is rarely the case. I switched from helm to bare k8s manifest for the same reason, helm/operator can be helpful, but not when the maintainer introduces breaking change (that does not relate to the service by itself) that needs a delete/deploy of the k8s resources

To be fair Helm is just a template layer for manifests, so it would take quite a bit of effort to break things.. especially if you're using a projects official Helm charts.

Yep, this is also the reason I don't use helm in any of my clusters. Too much magic and extra dependencies for a marginal (IMO) benefit.

An operator could trivially create the pvc ahead of any sts scaling op. The sts will simply look for an existing pvc by a predictable name and use it rather than create a new one, regardless of whether the pre-created pvc matches the spec of what the sts defines.

The flaw is definitely most acute when it is combined with operators, but it would make storage resizes vastly more accessible for less capable devs if it were able to be done declaratively via the spec instead of via a painful workaround.

Best operators are the operators that you don't have to deal with. The moment you have to investigate stuff, life is hell.

I like the idea of operator pattern (Operators aren't a built-in resource yet; they REALLY should be) but absolutely HATE how opaque they are. You really don't have a window into seeing what they are doing; often times (in my experience), you only know things are off much later on (because most operators rely on Kubernetes events to notify human operators that things have gone wrong).

Reply on news.ycombinator.com