I'd like to see more emphasis on UX for v2 for the most common operations, like deploying an app and exposing it, then doing things like changing service accounts or images without having to drop into kubectl edit.
Given that LLMs are it right now, this probably won't happen, but no harm in dreaming, right?
Even Terraform seems to live on just a single-layer and was relatively straight-forward to learn.
Yes, I am in the middle of learning K8s so I know exactly how steep the curve is.
Much of the complexity then comes from the enormous amount of resource types - including all the custom ones. But the basic idea is really pretty small.
I find terraform much more confusing - there’s a spec, and the real world.. and then an opaque blob of something I don’t understand that terraform sticks in S3 or your file system and then.. presumably something similar to a one-shot reconciler that wires that all together each time you plan and apply?
There’s the state file (base commit, what the system looked like the last time terraform succesfully executed). The current system (the main branch, which might have changed since you “branched off”) and the terraform files (your branch)
Running terraform then merges your branch into main.
Now that I’m writing this down, I realize I never really checked if this is accurate, tf apply works regardless of course.
I don't know how to have a cute git analogy for "but first, git deletes your production database, and then recreates it, because some attribute changed that made the provider angry"
You skipped the { while true; do tofu plan; tofu apply; echo "well shit"; patch; done; } part since the providers do fuck-all about actually, no kidding, saying whether the plan could succeed
Declarative reconciliation is (very) nice but not irreplaceable (and actually not mandatory, e.g. kubectl run xyz)
You can invent a new resource type that spawns raw processes if you like, and then use k8s without pods or nodes, but if you take away the reconciliation system then k8s is just an idle etcd instance
Imagine if pods couldn't reach other and you had to specify all networks and networking rules.
Or imagine that once you created a container you had to manually schedule it on a node. And when the node or pod crashes you have to manually schedule it somewhere else.
CRs for example. CRs are an amazing design concept. The fact that everything is a CR, with some exceptions (like everything in the v1 API) and their controllers are always Pods running somewhere is incredible. Debugging cluster irrops is so much easier once you internalize this.
However, this exemplifies my desire for better UX. It would be great if "kubectl logs" accepted an CR kind, which would have the API server would automatically find its associated controller and work with the controller-manager to pump out its logs. Shit, I should make that PR, actually.
Even better would be a (much improved) built-in UI that UIops people can use to do something like this. This will become extremely important once the ex-VMware VIadmins start descending onto Kubernetes clusters, which will absolutely happen given that k8s is probably the best vCenter alternative that exists (as non-obvious as that seems right now, though I work at Red Hat and am focused on OpenShift, so take that with two grains of salt)
I agree with you on a human level. Operators and controllers remind me of COM and CORBA, in a sense. They are hightly abstract things, that are intrinsically so flexible that they allow judgement (and misjudgement) in design.
For simple implementations, I'd want k8s-lite, that was more opinionated and less flexible. Something which doesn't allow for as much shooting ones' self in the foot. For very complex implementations, though, I've felt existing abstractions to be limiting. There is a reason why a single cluster is sometimes the basis for cell boundaries in cellular architectures.
I sometimes wonder if one single system - kubernetes 2.0 or anything else - can encompass the full complexity of the problem space while being tractable to work with by human architects and programmers.
You seem to want something like https://skateco.github.io/ (still compatible to k8s manifests).
Or maybe even something like https://uncloud.run/
Or if you still want real certified Kubernetes, but small, there is https://k3s.io/