To me this is just another hype train, like TensorFlow/Caffe before it, and like Hadoop/Spark before that.
These tools all have their uses in specific, non-common cases, but for your average business they are a net loss in time and money.
In my experience building and running Kubernetes clusters for Fortune 500 companies is that actually it's much simpler, it's just different from traditional infrastructure.
Different != Complex.
A vanishingly small number of use cases require 100's of nodes, and the accompanying orchestration. Stack Overflow runs a top 25 website on 9 webservers, and most of those are just redundancy. Nothing Ansible/Bash or Powershell/MSDeploy can't solve in orders of magnitude less time. And debugging/diagnosing problems is much easier.
Trying to integrate a k8s cluster into legacy infrastructure is non-trivial, and requires knowledgeable operations teams to deploy. Troubleshooting k8s requires extensive domain knowledge, in addition to foundational understanding of fairly advanced networking and distributed systems concepts. In my experience, working with k8s also requires much more vigilant watching of the project space for (breaking) changes, updated docs, SIG news, and github issues. Even if Google or Amazon is managing the k8s management, troubleshooting your app using kubectl, etc. is a new skill to learn.
I actually like a lot about using k8s, but I understand when people complain about the complexity.
K8S forces people to use an algebra, the expression of that algebra can take many forms. Containers are _just_ trees of processes with ... contained separation boundaries.
Currently running a application on anything is always complex in any kind of sense.
"You just take your app and put it in a container, then put that container in a pod, which is in a replica set with some other apps in their own containers, which is on a node in a cluster in a datacenter somewhere! Oh and the replica set is amongst a deployment. And if you want to talk to the outside world you'll need a sidecar with ngnix which is an app in a container in your pod. Or your deployment. But it's different than your app. Time to get kubectl and minikube installed..." People act like all these concepts are taught in beginner CS classes.
Kubernetes has some great attributes, but being simple isn't one. I feel sometimes that Google really pulled one on us, and this is just training people to treat the entire system as a black box, locking them in to services only top tier cloud companies have the expertise to provide and maintain reliably.
Only a few people need to know all the details. We have many developers that know absolutely nothing how this stuff works. They only know that they push code, and a container falls off the other side, which can even be installed automatically.
Also, you don't need sidecars. Add a k8s ingress, or even deploy a separate nginx that will do this for you (we use the second option). Developers do not care.
I'm starting to think that minikube is an anti-pattern.
I think you’re overestimating complexity based on lack of familiarity.
You do not have to use CoreOS, you do not need PXE boot. That's one of the possible ways.
What you do need is a handful of containers, and an overlay network. Which, if you do not want to manage, you can sign up for something like Google Cloud and let them manage it for you.