Hacker News Reader

Comment by kimbernator | Hacker News Reader

I didn't really enjoy my experience with the few operators I've worked with, mainly because they require the maintainer to build in some sort of access to basic kubernetes functionality. I see the benefit of operators, but I hated that in order to do something as simple as define memory/CPU limits to certain containers I would need to open a PR to the repo and wait weeks, sometimes months, for a new release.

It's frustrating to be a kubernetes admin but not have access to basic configuration options because the maintainers of even some very high-profile operators (looking at you, AWX) neglected to build in access to basic functionality.

View on HN

This is a common frustration of mine as well!

In the latest release of the spicedb-operator[0], I added a feature that allows users to specify arbitrary patches over operator-managed resources directly in the API (examples in the link).

There are some other projects like Kyverno and Gatekeeper that try to do this generically with mutating webhooks, but embedding a `patches` API into the operator itself gives the operator a chance to ensure the changes are within some reasonable guardrails.

[0]: https://github.com/authzed/spicedb-operator/releases/tag/v1....

Adding the patch api is neat! I’ve solved this in the past by embedding the entire PodSpec etc into the CRD

Did you call your CRD "Deployment"?

It’s a common pattern [1]. Other than the GP patch example, how else can users override the child objects your controller creates?

1 https://github.com/prometheus-operator/prometheus-operator/b...

I'm not saying they shouldn't customize their PodSpec, I'm saying if they do, your controller is exactly the same as the built-in Deployment controller, running user-provided pods. Why build a copy of it?

Because it’s an aggregate of a number of different resources types and custom logic not exactly the same as deployment.

I can see you don’t like operators and that’s fine. Many folks find it’s a useful pattern to follow

I might have to borrow that! Very clever

The SpiceDB operator looks like a prime example of something that should have been a Helm Chart. Migrations can be run in the containers.

Operators are just the non-containerized daemons of the Kubernetes OS. We did all this work to run everything in neatly encapsulated containers, and then everyone wants to run stuff globally on the whole cluster. What's the point? Do we just containerize clusters and start over?

I'm not sure what you're on about. Operators don't need to run in cluster at all. And even then, they can absolutely run as containers. And as far as permissions go, that's up to you. They're just regular service accounts.

Custom resource definitions are global to the cluster, and operators are usually written to watch those resources in the whole cluster.

You run the operator itself in a pod, but you can only have one of them running at a time, hence they are global. You can run as many Alpine/Ubuntu/... containers as you want, and as many ElasticSearch or PostgreSQL as you want, in isolation, that's the beauty of containers. But you can only run one SpiceDB operator or one ElasticSearch operator, on the whole cluster. We're back to square one.

I don't think it's quite as bad as you're suggesting. Operators are control-plane, your other examples are data-plane. You can also only run one Deployment controller in a kube cluster, but many Deployments. And you can run many controllers for a single CRD, see the Gateway APIs for example, but it's not as common and is more work.

That said, I do think the direction that Kube has gone with CRDs, by leaning into making them more like built-in apis instead of allowing them to be differentiated as an external extension point has limited what you can do with them.

When ThirdPartyResrouces just carved out a group and a name in the API it was simple to have multiple copies and multiple versions running together in a cluster. But that has become more difficult with time, especially with ConversionWebhooks - a feature that forces all controllers for an API to agree not just on a version, but on the specific ways versions convert into each other.

I get the sentiment. We held off on building an operator until we felt there was actually value in doing so (for the most part, Deployments cover the operational needs pretty well).

Migrations can be run in containers (and they are, even with the operator), but it's actually a lot of work to run them at the right time, only once, with the right flags, in the right order, waiting for SpiceDB to reach a specific spot in phased migrations, etc.

Moving from v1.13.0 to v1.14.0 of SpiceDB requires a multi-phase migration to avoid downtime[0], as could any phased migration for any stateful workload. The operator will walk you through them correctly, without intervention. Users who aren't running on Kubernetes or aren't using the operator often have problems running these steps correctly.

The value is in this automation, but also in the API interface itself. RDS is just some automation and an API on top of EC2, and I think RDS has value over running postgres on EC2 myself directly.

As for helm charts, this is just my opinion, but I don't think they're a good way to distribute software to end users. The interface for a helm chart becomes polluted over time in the same way that most operator APIs become polluted over time, as more and more configuration is pulled up to the top. I think helm is better suited to managing configuration you write yourself to deploy on your own clusters (I realize I'm in the minority here).

[0]: https://github.com/authzed/spicedb/releases/tag/v1.14.0

> I would need to open a PR to the repo and wait weeks, sometimes months, for a new release.

Just curious, is this a limitation of the Operators framework, or that of your system's implementation? My knee-jerk reaction is that any implementation should absolutely not require opening ticket. After all, Amazon's API mandate happened 20 years ago, and Netflix followed suit to achieve phenomenal productivity for their engineers. I have a hard time imagining why any engineer would think that gatekeeping configuration with PR is a good idea(a UI with proper automation and approval process that hides generated PR for specific use cases is a different matter)

Not a kubernetes expert, but my understanding is that that operators are regular programs that run in a kubernetes container and interact with the kubernetes API to launch/manage other containers and custom kubernetes resources.

An operator (or its custom resource) can be configured by Kubernetes YAML/API and its upto the creator of the operator to specify the kind of configuration. If the operator creator did not specify options to set cpu/memory limits on the pods managed by the operator, then you can't do anything. You have to add that feature into the operator and then make a pull request and wait for it to be upstreamed.

Or fork it instead. Same thing for helm charts (except forking and patching them is easier than forking an operator).

Reply on news.ycombinator.com