Glad I'm not the only one. I'm sure I'm not the smartest engineer/sysadmin in the world, but I'm also not the dumbest and I have never gotten an on-premises Kubernetes installation to work.
The way I manage containers is lxc and shell scripts. I understand it, and it works.
Don't tell the boss or the customers, but most of the time when release notes for a new version come out, I look at them and go "WTF, why do we need that, I better do some research." It's fast changing, complex as hell, and absolutely brutal. That said most things are there for a reason and once I dig in I usually see the need.
That said I do love it despite its warts. There's no doubt some Stockholm Syndrome at play here, but I love the API (which is pretty curlable btw, a mark of a great API IMHO) and the principles (declarative, everything's a YAML/JSON object in etcd, etc). I see it the same way I did C++ (which I also loved). It gives you great power which you can use to build an elegant, robust system, or you can create an unmaintainable, complex, monster of a nightmare. It's up to you.
[0] https://goteleport.com/blog/kubernetes-release-cycle/ , HN discussion: https://news.ycombinator.com/item?id=16285192
What is deadly difficult is getting networking to work. Even a comparably "easy" thing with a couple of two-NIC machines (one external, internet-routable, one DMZ) cost me a fucking week.
What's even worse is when one has to obey corporate restrictions - for example, only having external interfaces on "loadbalancer" nodes:
- First of all, MetalLB only has one active Speaker node which means your bandwidth is limited to that node's uplink and you're wasting the resources of the other loadbalancers.
- Second, you can taint your nodes to only schedule the MetalLB speaker on your "loadbalancer" nodes via tolerations... but how the f..k do you convince MetalLB to change the speaker node once you apply that change?!
- Third, what do you do when you want to expose N services but only have one or two external IPs? DC/OS was way more flexible, you had one set of loadbalancers (haproxy) that did all the routing, and could run an entire cluster on four machines - two LBs, one master, one worker. There is no way to replicate this with Kubernetes. None.
It's not Kubernetes but it doesn't try to be at all.