Sooooooo I'm about to basically go off
> To be fair, OSBAPI is a bigger upfront effort for a service author than fiddling with kubebuilder and sorta-kinda doing some basic service-y stuff yourself. Plus CRDs are the cool thing right now and I expect they will continue to build in momentum.
kubebuilder and the whole "ecosystem" for building controllers is unnecessarily hard to use and ideally shouldn't have been this way in the first place. Writing an operator is more painful than it should be, and I think many of the engineering decisions early on are what made it this way.
IMO CRDs are actually the defining feature of k8s but no one realized it until later. They set out to build a cathedral when they should have been building a more orderly bazaar.
> Where it will come back to bite everyone is that every service will have its own Operator and its own CRDs and these will often be different in shape, behaviour and conventions than other Operators and CRDs. It'll be like the bad old days of every piece of software shipping with its own pile of bash scripts for management and twenty different process managers. You know: the sort of thing Red Hat helped to bring an end to.
Right, but maybe this time people will solve that problem the right way -- by standardizing on the representation (CRD) in this case. I mean this in the general sense -- the case I think of is trying to capture all the different ways you can containerize something as they evolved over the years:
- LXC exists (let's call this an `ubuntu.com/Container` entity)
- Docker is introduced, people go wild (let's call this a `docker.com/Container` entity)
- OCI container spec gets written (Let's call this an `opencontainers.org/Container` entity)
- People want to just be able to make a "Container" that runs on whatever operator is available ==> They make a `foo.com/Container` that represents a the lowest common denominator
This kind of progression is the only way you can keep your rapid iteration but have consistent behavior -- foo.com takes the bullet to provide and maintain consistent behavior while the other container types can iterate as fast as they desire.
> After we reach the point where you're spinning up whole clusters because the PostgreSQL operator doesn't play nice with the Kafka operator on Tuesdays when the breeze is too strong, someone wise and brave and famous on twitter will yell "this is insanity! Surely we can rationalise this into a single API, a single mechanism!"
This I think is a symptom is an orchestrator/operators that aren't robust enough -- operators should not be able to take out other operators, except through very well defined integration points. Hard to write the interface in practice but I think we can do better than what kubernetes currently is.
I've been sitting on the concept for a kubernetes competitor that (I think) is simpler but haven't gotten around to writing it. Here are the broad strokes:
- The only pattern is provider/resource (essentially what k8s knows as CRDs and operators)
- Complexity increases only by composition (i.e. you want to write a `KafkaProvider`? looks like you'll need a `ContainerProvider`, `NetworkProvider`, etc)
- No GRPC (good 'ol HTTP1.1, upgradable to 2/3 in the standard web-compatible ways, with "Content-Type" if you really feel like you need to squash your request/responses, and SSE if you need to stream stuff)
- jsonschema + hyperschema + JSON-LD for automated everything (verification, request building, semantics parsing)
- Rust
- Single binary deployment (as in get your VM, maybe set some kernel flags, put a single binary on it, run that binary, and your cluster is up).
I've got pages of notes on what this could be but at this point it's vaporware -- Who knows if I'll ever get to work on this idea but I sure wish someone would. I don't think there are any companies willing (essentially, dumb enough) to try to compete with Kubernetes so I don't think anyone is even considering trying to build a new but refined k8s.