Kubernetes gets a lot of shade, and rightfully so. It’s a tough problem. I do hope we get a Ken Thompson or Rich Hickey-esque solution at some point.
k8s is complex not unnecessarily, but because k8s is solving a large host of problems. It isn't JUST solving the problem of "what should be running where". It's solving problems like "how many instances should be where? How do I know what is good and what isn't? How do I route from instance A to instance b? How do I flag when a problem happens? How do I fix problems when they happen? How do I provide access to a shared resource or filesystem?"
It's doing a whole host of things that are often ignored by shade throwers.
I'm open to any solution that's actually simpler, but I'll bet you that by the time you've reached feature parity, you end up with the same complex mess.
The main critique I'd throw at k8s isn't that it's complex, it's that there are too many options to do the same thing.
Unfortunately unless you've got a lot of k8s experience that scale/complexity lower bound isn't super obvious. It's also possible to have your scale/complexity accelerate from "k8s isn't worthwhile" to "oh shit get me some k8s" pretty quickly without obvious signs. That just compounds the TMTOWTDI choice paralysis problems.
So you get people that choose k8s when it doesn't make sense and have a bad time and then throw shade. They didn't know ahead of time it wouldn't make sense and only learned through the experience. There's a lot of projects like k8s that don't advertise their sharp edges or entry fee very well.
Maybe compared to Heroku or similar, but compared to a world where you're managing more than a couple of VMs I think Kubernetes becomes compelling quickly. Specifically, when people think about VMs they seem to forget all of the stuff that goes into getting VMs working which largely comes with cloud-provider managed Kubernetes (especially if you install a couple of handy operators like cert-manager and external-dns): instance profiles, AMIs, auto-scaling groups, key management, cert management, DNS records, init scripts, infra as code, ssh configuration, log exfiltration, monitoring, process management, etc. And then there's training new employees to understand your bespoke system versus hiring employees who know Kubernetes or training them with the ample training material. Similarly, when you have a problem with your bespoke system, how much work will it be to Google it versus a standard Kubernetes error?
Also, Kubernetes is really new and it is getting better at a rapid pace, so when you're making the "Kubernetes vs X" calculation, consider the trend: where will each technology be in a few years. Consider how little work you would have to do to get the benefits from Kubernetes vs building those improvements yourself on your bespoke system.
If you have just a single monolith app (such as a wordpress app) then sure, k8s is overkill. Even if you have 1000 instances of that app.
It's once you start having something like 20+ distinct services that k8s starts paying for itself.
Are you referring to instances of your application, or EC2 instances? If instances of your application, in my experience it doesn't really do much for you unless you are willing to waste compute resources. It takes a lot of dailing in to effectively colocate multiple pods and maximize your resource utilization. If you're referring to EC2 instances, well AWS autoscaling does that for you.
Amazon and other cloud providers have the advantage of years of tuning their virtual machine deployment strategies to provide maximum insulation from disruptive neighbors. If you are running your own Kubernetes installation, you have to figure it out yourself.
> How do I know what is good and what isn't?
Autoscaling w/ a load balancer does this trivially with a health check, and it's also self-healing.
> How do I route from instance A to instance b?
You don't have to know or care about this if you're in a simple VPC. If you are in multiple VPCs or a more complex single VPC setup, you have to figure it out anyway because Kubernetes isn't magic.
> How do I flag when a problem happens?
Probably a dedicated service that does some monitoring, which as far as I know is still standard practice for the industry. Kubernetes doesn't make that go away.
> How do I fix problems when they happen?
This is such a generic question that I'm not sure how you felt it could be included. Kubernetes isn't magic, your stuff doesn't always just magically work because Kubernetes is running underneath it.
> How do I provide access to a shared resource or filesystem?
Amazon EFS is one way. It works fine. Ideally you are not using EFS and prefer something like S3, if that meets your needs.
> It's doing a whole host of things that are often ignored by shade throwers.
I don't think they're ignored, I think that you assume they are because they are because those things aren't talked about. They aren't talked about because they aren't an issue with Kubernetes.
The problem with Kubernetes is that it is a massively complex system that needs to be understood by its administrators. The problem it solves overlaps nearly entirely with existing solutions that it depends on. And it introduces its own set of issues via complexity and the breakneck pace of development.
You don't get to just ignore the underlying cloud provider technology that Kubernetes is interfacing with just because it abstracts those away. You have to be able to diagnose and respond to cloud provider issues _in addition_ to those that might be Kubernetes-centric.
So yes, Kubernetes does solve some problems. Do the problems it solves outweigh the problems it introduces? I am not sure about that. My experience to Kubernetes is limited to troubleshooting issues with Kubernetes ~1.6, which we got rid of because we regularly ran into annoying problems. Things like:
* We scaled up and then back down, and now there are multiple nodes running 1 pod and wasting most of their compute resources.
* Kubernetes would try to add routes to a route table that was full, and attempts to route traffic to new pods would fail.
* The local disk of a node would fill up because of one bad actor and impact multiple services.
At my workplace, we build AMIs that bake-in their Docker image and run the Docker container when the instance launches. There are some additional things we had to take on because of that, but the total complexity is far less than what Kubernetes brings. Additionally, we have the side benefit of being insulated from Docker Hub outages.
This begs to question if there is a wrong or right way of doing things and if a single system can adapt fast enough to the rapidly changing underlying strategies, protocols, and languages to always be at the forefront of what is considered best practices in all levels of development and deployment.
These unified approaches usually manifest themselves as each cloud providers best practice playbooks, but each public cloud is different. Unless something like Kuberenetes can build a unified approach across all cloud providers or self hosting solutions then it will always be overly complex because it will always be changing for each provider to maximize their interests in adding their unique services.
And then there are all of the different kinds of resources and the general UX problem of managing errors ("I created an ingress but I can't talk to my service" is a kind of error that requires experience to understand how to debug because the UX is so bad, similarly all of the different pod state errors). It's not fundamentally complex, however.
The bits that are legitimately complex seem to involve setting up a Kubernetes distribution (configuring an ingress controller, load balancer provider, persistent volume providers, etc) which are mostly taken care of for you by your cloud provider. I also think this complexity will be resolved with open source distributions (think "Linux distributions", but for Kubernetes)--we already have some of these but they're half-baked at this point (e.g., k3s has local storage providers but that's not a serious persistence solution). I can imagine a world where a distribution comes with out-of-the-box support for not only the low level stuff (load balancers, ingress controllers, persistence, etc) but also higher level stuff like auto-rotating certs and DNS. I think this will come in a few years but it will take a while for it to be fleshed out.
Beyond that, a lot of the apparent "complexity" is just ecosystem churn--we have this new way of doing things and it empowers a lot of new patterns and practices and technologies and the industry needs time and experience to sort out what works and what doesn't work.
To the extent I think this could be simplified, I think it will mostly be shoring up conventions, building "distributions" that come with the right things and encourage the right practices. I think in time when we have to worry less about packaging legacy monolith applications, we might be able to move away from containers and toward something more like unikernels (you don't need to ship a whole userland with every application now that we're starting to write applications that don't assume they're deployed onto a particular Linux distribution). But for now Kubernetes is the bridge between old school monoliths (and importantly, the culture, practices, and org model for building and operating these monoliths) and the new devops / microservices / etc world.