CRDs Killed the Free Kubernetes Control Plane

There are a number of reasons. One is that states don't stay simple for very long. Just the other day I saw someone ask "how do I find all the children of a parent resource?". The answer was to iterate over everything that might have that parent resource as an OwnerRef. In a normalised schema this would be simple. In a document database it is not.

Another problem: there are no multi-document transactions. If I rely on updating A and B together, I am out of luck. I will instead need to account for A and B being out of sync, which involves 4 permutations: A,B; A, !B; !A, B; !A, !B. As a CRD controller author I am now responsible for a shitty database. Where Kinds have crisp hierarchies this is fine, the supervising controller can inspect and repair the lower documents. But many things are not hierarchical in nature and you can easily wind up with control loop fights.

Another is that there's a difference between "I already have a cluster" and "I am using the API server as a database". Because CRDs wind up in a single global document store, folks inevitably want them to have hard isolation. RBAC, namespaces etc get you some of the way but it's basically just hard. Eventually some bright spark suggests putting things into a different cluster, at which point the balance of ease flips the other way.

CRDs are super useful extension points (note, for example, I am writing a book about Knative), but they are not suitable for most application and business logic purposes.

If your app is running on a modern computer anyway, why is it a bad idea to store some simple state in EFI variables?