The Missing Kubernetes Type System | Hacker News Reader

The Missing Kubernetes Type System

danielmangum.com

82 pointsingve22 comments

I've been a PM for Kubernetes at two of the major contributing companies and helped popularize Operators.

The project that will replace Kubernetes will be something that provides builtins that compose together to EXTEND the system. With such a foundation, you'll be given the flexibility to evolve over time and correct design flaws. Kubernetes was in a rush to beat competitors out in the early orchestration wars, so they focused on primitives that people needed to deploy "today". I think they did a good job, but the focus wasn't on extensibility.

In an ideal world (and I've proposed), there be a long term SIG/WG that experimented with custom controllers to add new primitives that focused on EXTENDING Kubernetes. Most companies at the time wanted to tighten up the core APIs, not realizing that they should be skating towards where the puck is going. Now every major Kubernetes provider is past the point of just getting clusters into folks' hands and are realizing they need to add more functionality, but have no means that doesn't come at the cost of explosive complexity.

This is exactly right. I don’t think it’ll happen within k8s because if you’ve ever worked with sig-api you’ll know they aren’t big on innovation.

The biggest problem with kube is the declarative interfaces aren’t enough for most problems. You need imperative methods as well and the only way to sort of do this is with aggregated API servers (yuck!)

Even the core Pod resource even has a /logs endpoint which is imperative.

Also kubectl is a mess, verb->noun makes no sense at all.

Out of curiosity, and particularly because I'm working on a Webassembly based alternative to Kubernetes:

What would you do differently if you could start from scratch?

Please, for the love of interop, do not use GRPC. Not every language supports it (JSON is way better anyway). There are a number of times I’ve wanted to build on something in k8s. But I’m not learning go just to write an api controller. I’m not going to write a GRPC implementation just to write an api controller in my language of choice either.

Full disclosure: I really hate GRPC as a wire protocol.

Worse than using a binary protocol and an IDL, I think, are the myths that gRPC proponents seem to believe: e.g. you can't delete fields and all fields are optional, ergo <voila> you now have guaranteed backwards compatibility.

When I used it for ruby. I was totally fascinated by it. Wow. It was a horrible experience. Lack of proper support made everyone hard.

Loved the concepts though.

Very curious where you see the advantage of using Webassembly there. Presumably this is not running in a browser.

Webassembly is also gaining steam for server workloads due to several advantages (less overhead, better capability based security, composability, ...).

See Spin [1], Wasmcloud [2], Lunatic [3], etc.

My system is based on a distributed Webassembly runtime.

The reason for taking inspiration from Kubernetes is making deployment of distributed workloads on that runtime easy.

A nice benefit for a Kubernetes alike system is that the equivalent to operators and controllers can be much more light-weight WASM actors that are easier to develop, deploy and scale.

[1] https://github.com/fermyon/spin

[2] https://github.com/wasmCloud/wasmCloud

[3] https://github.com/lunatic-solutions/lunatic

Webassembly as a desirable server side technology... I suppose I need to recontextualize what I understand webassembly to be. Hmm.

It's like the JVM but more languages can run on it and it has buy in from more companies. Imagine all the promises of Java in the 90s/2000s but the open source community is actually interested in it.

Is that actually true though, or is this more of an aspirational goal for WASM? The JVM runs an absolute ton of different languages.

Even better (and I expect you're aware of this already), you can run Webassembly workloads under Kubernetes control:

https://krustlet.dev/

Yeah, krustlet is great for integrating into existing Kube deployments. I'm just tackling the problem from the other side.

I'm paid to work on a FOSS operator, and I couldn't quite discern a) the issue and b) the proposed solution.

Is the author wanting to move away from K8s' Go style object orientation (structs and functions that act on them) to more of a Java style OO where objects express both data and behaviour?

Couldn’t tell either. I’m familiar with the conceptual building blocks, but I‘d have a hard time extracting a tl;dr.

Fully agree! Here’s my post on this problem space from nearly a decade ago: https://kyle.marek-spartz.org/posts/2014-05-08-your-service-...

That's a nice formalization of services. Did you try implementing it?

I'm not sure the analogy with types is at all helpful in explaining this.

If I understand the author right, the critique is that Kubernetes wants to present a unified, declarative model for arbitrary orchestration behavior. To do this, it uses controllers/operators/CRDs/etc, but the code within these controllers is often some of the imperativiest-imperativing stuff you've ever seen. This is "fine" for some core primitives, but as you demand more complex orchestrations out of k8s it's more likely you have to write these yourself, and it's messy and "hard", beyond what most teams can or want to invest, because of that.

(Up to this point, I agree fully.)

But then the "types" analogy comes to the fore, and we end up with a system that's even more complex, a kind of HKT for declarative behaviors.

I'm not sure it really solves the implementation problems to begin with, but even if it does - I predict the teams struggling to implement custom operators are going to slam into this new wall even harder trying to model their operational processes like this.

How long until we finally admit a lot of things an operator (human) does are imperative, and therefore the k8s "declarative only" model is fundamentally broken? What we need is more carefully-chosen imperative escape hatches, not more API options.

I appreciate your summary. I wonder if something like metacontroller aligns with this problem space.

https://metacontroller.github.io/metacontroller/intro.html

>"But then the "types" analogy comes to the fore, and we end up with a system that's even more complex, a kind of HKT for declarative behaviors."

I agree with all of your points. What is HKT here though?

Not parent but most likely Higher-kinded Type[0]

[0]: https://en.m.wikipedia.org/wiki/Kind_(type_theory)

What we need is an API language that can do this elegantly. GraphQL has interfaces, something roughly like:

type Foo implements DeclarativeObject

would work, then you could always add methods on top

Reply on news.ycombinator.com