You’re right that kubernetes is a bit batteries included, and for that its tempting to take it off the shelf because it “does a lot of needed things”, but you don’t need one tool to do all of those things.
It is ok to have domain specific processes or utilities to solve those.
* your stack almost always ends up closely tied to one cloud provider. I've done and seen cloud migrations. They are so painful and costly that they often just aren't attempted.
* Cloud services make it much harder to run your stack locally and on CI. There are solutions and workarounds, but they are all painful. And you always end up tied to the behaviour of the particular cloud services
> but you don’t need one tool to do all of those things
To get the same experience, you do. And I don't see why you would want multiple tools.
If anything, Kubernetes isn't nearly integrated and full-featured enough, because it has too many pluggable parts leading to too much choice and interfacing complexity. Like pluggable ingress, pluggable state database, pluggable networking stack, no simple "end to end app" solution ( KNative, etc), ... This overblown flexibility is what leads to most of the pain and perceived complexity, IMO.
Huh, I guess you are spot on. My first experience with kubernetes was k3s and I couldn't for a long time figure out what's all the fuss is about and where is all that complexity people talk so much about. But then I tried vanilla kubernetes.
Kubernetes is currently the best way to wrap up workloads in a cloud agnostic way. I've written dozens of services for K8s using different deployment mechanisms (Helm, Carvel's kapp, Flux, Kustomize) and I can run them just as easily in my home K8s cluster and in GCP. It's honestly incredible; I don't know of any other cloud tech that lets me do that.
One thing I think a lot of people miss too, is how good the concepts around Operators in Kubernetes are. It's hard to see unless you've written some yourself, but the theory around how operators work is very reminiscent of reactive coding in front end frameworks (or robotics closed loop control, what they were originally inspired by). When written well they're extremely resilient and incredibly powerful, and a lot of that power comes from etcd and the established patterns they're written with.
I think Kubernetes is really painful sometimes, and huge parts of it aren't great due to limitations of the language it's written in; but I also think it's the best thing available that I can run locally and in a cloud with a FOSS license.
I'll give you the benefit of the doubt here and say that some of the basics are indeed cloud agnostic.
However, it's plain for many or most to see that outside of extremely "toy" workloads you will be learning a specific "flavour" of Kubernetes. EKS/GKE/AKS etc; They have, at minimum, custom resource definitions to handle a lot of things and at their worst have implementation specific (hidden) details between equivalent things (persistent volume claims on AWS vs GCP for example is quite substantially different).
Thinking about it; the things I see as very cloud agnostic: Horizontal pod autoscaling, Node autoscaling, Layer 4 loadbalancing, Persistent volumes, Volume snapshots, Certificate managment, External DNS, External secrets, Ingress (when run in cluster, not through a cloud service),
That ends up covering a huge swath of my usecases, probably 80-90%. The main pain points I usually run into are: IAM, Trying to use cloud layer 7 ingress (app loadbalancers?)
I totally agree the underlying implementation if resources can be very different, but that's not the fault of Kubernetes; it's an issue with the implementation from the operator of the K8s cluster. All abstractions are going to be leaky at this level. But for PVCs I feel like storageclasses capture that well, and can be used to pick the level of performance you need per cloud; without having to rewrite the common provision of block device.
When I say something is easy to move, I mean that when I build on top of it, it's easy for users to run it in their cloud of choice with changes in config. It also means I have flexibility with where I choose to run something after I've developed it. For example I develop most stuff against minikube, then deploy it to GCP or a local production k8s. If I was using Terraform I couldn't do that.
If you'll excuse a slight digression, but I think there's a tendency atm to rather pay $1 in extra complexity a 100 times over time, than pay a $5 one-time fee. Like if repeating something similar twice - even if it's easy and not really a lot of effort - is a sign of failure and thus unbearable.