Kubernetes 1.12
kubernetes.io
kubernetes.io
Zalando (disclaimer I work there, though not on K8S infrastructure. I just deploy to K8S) does upgrades this way: https://m.youtube.com/watch?v=1xHmCrd8Qn8
So every 3 months is too fast for a lot of teams (again, not technical issue, this is culture of enterprise) so they may put off an upgrade. Once you do that the upgrade becomes more risky because there is obviously more change from 1 to 3 than 1 to 2.
What I'm describing is not an issue specific to k8s, this is just a general enterprise infrastructure issue and one of the many reasons enterprises like LTS releases.
Unfortunately this is difficult because most distributions don't keep up with the release cadence.
One of (the most?) popular way to deploy Kubernetes is the Kops project. As far as I know there is nobody who is paid to work on Kops, thus the bugs and PR's don't get enough hours invested in them.
Distribution and packaging is very boring and fiddly grunt work, it would be wonderful if somehow we as a community could figure out how to fund full time work on a distribution in the OSS space. Perhaps CNCF could do a better job of funding this somehow.
It's a hard problem, because the organizations most suited to solving it are CoreOS/RedHat/Microsoft/Google/Amazon/etc, and they have a vested interest in putting that effort into making their hosted solutions a great experience instead.
Kubeadm looks like a great step in the right direction, but as far as I'm aware, there aren't any production quality distributions for installing clusters with it yet.
Having your entire k8s infrastructure go down due to a "oopsie" in an upgrade is a risk. And mitigating these risks takes time and effort.
I hearken back to a snippet from a recent article:
> 2014: We must adopt microservices to solve all the problems with monoliths
> 2016: We must adopt Docker to solve all the problems with Microservices
> 2018: We must adopt Kubernetes to solve all the problems with Docker
What's next? "We must adopt Rancher to solve all the problems with K8s"?
If you use the managed hosting/cloud providers then updates are also automatic and easy.
A point of reference, Amazon EKS only runs 1.10 which will not be supported by the community in 3 months. So not all managed services are really keeping up.
How many of the K8S vendors are selling distributions vs support plans? Or are they always entangled? And if the community did do LTS, isn't that just shifting the work to open-source maintainers rather than focused and paid vendors?
As far as I was aware of all the on-prem options cleave to the main project support lifecycle...
Red Hat provides support our Kubernetes distribution OpenShift (upstream is OKD [formerly called Origin] project). We have an LTS release being cut soon as well. Watch OpenShift Commons for more info.
I realize that the aim is mostly at the enterprise market who don't run autoscaling groups or cattle-grade servers, but you're scaring off potential clients with these kinds of setups.
If you have a specific suggestion/RFE, I’d be happy to push on it.
A technical solution: A service that counts nodes or available VCPUs and reports it for billing. Kubernetes metrics makes this really simple.
A contract solution: Realize that bare metal is not the future (even for enterprises), and adopt a different billing model; following Oracle's per-CPU model is not the the way to be a leader in this industry. Per cluster, by bands of service counts, per site, per hour with minimum hour counts, etc.
that said, per core pricing is bullshit. and for exactly the reasons you mentioned.
What's the support lifecycle going to be on that?
Each one I’ve done on our cluster over the past couple of years has caused something to break, often in a strange and obscure manner.
One time an automatic master upgrade happened at 9pm and effectively sent all of our logs into a black hole for reasons unknown. As with many of the other upgrade hiccups, I just had to poke, prod, delete and recreate entities and things magically started working again.
There is so much going on under the hood it makes sense that there would be issues with upgrades, so the frequency of the releases is frustrating. Dealing with after hours alarms just because the k8s release cycle is so aggressive really grinds my gears. I would fully support an LTS version.
upgrades don't get easier when you delay them and make them bigger, they get harder. if you think k8s upgrades are bad now, having them change twice as much half as often would be worse.
Having an LTS release available offers the flexibility to choose when to upgrade based on any number of factors affecting the business and engineering cycles, the same way we pin software libraries to specific versions so our own stability and build consistency isn't subject to the varying release cycles of the package authors.
There's definitely a balance to be found between waiting so long to upgrade that it becomes insurmountable, and having to frequently run disruptive, mystery-downtime-causing upgrades for unneeded new features.
In proper CI/CD practices, build "stability" is an anachronism. Master is always green. Test changes before merging. If the changes fail the test suite then don't merge and don't deploy until it's fixed. Master stays green. Why is dynamic infrastructure any different?
LTS only makes sense as a strategy if testing and carrying out the upgrade involve significant amounts of work which needs to be planned out in advance. Ideally, this isn't the case.
Errors have never been major for us though, and they do seem to be greatly reduced recently with no noticeable issues in the 1.10x line.
And still it's a bit of a battle to get people to test new versions of things, k8s included. "But it's working" is as much of a rallying cry as it ever was, "cloud" or no.
Between company policy on social media and internal weirdness I can't really write a GitHub or Medium contribution to HN that anyone would want to consume anyway.
However I do plan to document what I've learned in a generic sort of "If you use Digital Ocean, this is how to do X, Y, Z" way.
Not using DigitalOcean currently, but look forward to reading that someday.
- get a local test setup with minikube (https://github.com/kubernetes/minikube) and Helm (https://www.helm.sh/) and try out a few applications - convert a project of yours to a Kubernetes deployment, then a Helm chart
If both steps are successful and you still feel like Kubernetes is doing a better job for you then Heroku, get a Kubernetes professional. I promise you, there will be as many problems as with any other environment. The only difference is that the Kubernetes issues are not documented.
Move that one to the icebox then!
They feel it's way more complicated and cumbersome and say they don't want to write thousand line YAML files and YAML engineering isn't their job.
Also, the pipelines Heroku provides and new CD is far superior. They don't want to reinvent Heroku.
Personally after I got k8s I felt about it how I felt about git. I think it's game changing and everyone should learn something about it. Can understand the frustration though.