Kubernetes Needs an LTS
matduggan.com
matduggan.com
Software is a garden that needs to be tended. LTS (and to a lesser extent requirements for large amounts of backwards compatibility) arguments are the path the ossification and orgs running 10+ year out of date, unsupported legacy garbage that nobody wants to touch, and nobody can migrate off because it’s so out of whack.
Don’t do this. Tend your garden. Do your upgrades and releases frequently, ensure that everything in your stack is well understood and don’t let any part of your stack ossify and “crust over”.
Upgrades (even breaking ones) are easier to handle when you do them early and often. If you let them pile up, and then have to upgrade all at once because something finally gave way, then you’re simply inflicting unnecessary pain on yourself.
> out of date, unsupported legacy garbage
I completely agree, but that’s also not-incompatible with my argument.
Some of the attitudes in here about software needs to be constantly evolving is just odd. Many many systems run on legacy software and hardware because, for the most part, they just work.
Innovation and evolution are important, yes. But churn for the sake of churn does not fit every (most) use case...
You can't see me, but I'm violently nodding in agreement. Faithfully adhering to these best practices isn't always possible, though; management gonna manage how they manage and now how your ops team wants them to manage.
> Upgrades (even breaking ones) are easier to handle when you do them early and often. If you let them pile up, and then have to upgrade all at once because something finally gave way, then you’re simply inflicting unnecessary pain on yourself.
How different could things be if k8s had a "you don't break userland!" policy akin to the way the Linux kernel operates? Is there a better balance between new stuff replacing the old and never shipping new stuff that would make more cluster operators more comfortable with upgrades?
If we only think of open source in datacenters we limit our thinking to just one of the many places it's used.
Seriously, I am sick of elevators thay run Javascript, cars that run docker and self driving cars running linux.
This is not good enough. Our industry is a joke.
While I do not appreciate everything being a Web app it's also a very robust platform to build upon, there aren't many projects that gets as many eyes as browsers and their components.
[0]: https://www.i-programmer.info/news/149-security/8548-reboot-...
exactly, that's horrifying. Only seppuku would be enough of a punishment.
Mechanical things need mechanical software that never fails.
Now, the embedded industry has largely failed at software development and they are using softeare that was not meant for them.
Recently worked with a team doing edge architecture and design that has had at least 5 iterations of k8s design and vendors. Why is that? What was the solution?
My opinion at the end (even though I love cloud managed k8s for commodity/insecure in the commercial space) is not to use k8s for edge and secure. Stop pretending. It's a mess and it won't get better.
Manufacturing comes to mind, shuttering a machine down to apply patches monthly is going to piss off the graph babysitters, especially if the business is a 24/7 operation, and most are currently.
In an ideal world there would be times every month to do proper machine maintenance, but that doesn't translate to big money gains for the glutton of shareholders who don't understand anything, let alone understand that maintenance prolongs processes as opposed to running everything ragged.
> if the business is a 24/7 operation
> In an ideal world there would be times every month to do proper machine maintenance
Then they find a compatability problem and the car n3eds to be rebuilt
It's mainly good for keeping high paid people employed, not keeping your servers stable.
I ran a cluster 1.5 years in production, took me so much energy. Especially that one night where digital ocean managed cluster forced an update that crashed all servers; and there was no sane way to fix it.
I'm back to stability with old school VPS; it just works. Every now and then you run a few patches. Simple & fast deploys; what a blessing.
I use a managed instance and manage my own private instance.
They're not saying they moved to running k8s on a VPS - they're saying they moved to using a VPS instead of k8s (i.e. have escaped the k8s upgrade cycle).
More likely, you will be constrained by a patchwork quilt of contracts with customers and suppliers, or by regulatory and statutory requirements, or technology risk policies, and to get approval you’ll need to schedule end-to-end testing with a bunch of stakeholders whose incentives aren’t necessarily aligned to yours.
That all adds up to $$$, and that’s why there’s a demand for stability and LTS editions.
If you automate everything you will never have an issue with K8, you can deploy all the required dependencies in one go. You can run the tests, if done correctly this literally requires 10 minutes.
This argument to have an LTS Version for me is like air. It's the kind of Nuclear Reactor argument. We did have the money to buy Uranium to burn but now we can't afford to dispose it normally.
Kubernetes is for flexibility not for big companies which want to use their software development processes of the 90's...
If all that holds, my question then becomes, why does your system not have a preprocesisng and postprocessong layer for compatibility? Stuff like that is easier than ever to build, and would allow your components to grow with the ecosystem?
But many interesting useful real-world systems are difficult to contain within a perfect black box. Abstractions are leaky. An API gateway, for example, cannot hide increased latency in the backend.
People accountable for technology have learned, through years of pain, not to trust changes that claim to be purely technical with no possible impact on business functionality. Hence testing, approval and cost.
I trial upgraded a years old Django project today to 5.0 and it took almost zero work. I hadn’t touched versions (other than patch)in over a year. That’s the way I want it. Admittedly this was less about an LTS and more about sensible design with upgrading in mind.
Django is great at backwards compatibility, but to be honest they haven't added many revolutionary features to Django for years, except a half-baked implementation of async support.
LTS is a commitment. That is all. If someone is uncomfortable with such a commitment, then that's fine, let me free market sort it out. But what LTS does is it tells everyone (including paying customers) that the formats/schemas/APIs/etc.. in that version will be supported for a very long time and if I adopt it, I won't have to think about it or budget too much for its maintenance for a period of time measured in months/years.
I would go the extra mile here and say that offline format should be supported FOREVER. None of that LTS bs for offline data, ever. LTS means that you're accountable for some of the costs in your partnership with the customers. If you move fast and break things, they will have to work extra just to keep up. If you move fast but mindful with back-compat, you will work extra but your customer will be happier. That is all.
Re-reading your comment gives me chills and reinforces my belief that I will never pay money to Google (they have a similar gung-ho attitude against 'legacy garbage') or have any parts of my business depend on stuff which reserves the liberty of breaking shit early and often.
We had very few control over the customers stuff
We did not had to : avoid breaking changes (or impose them) and transparent upgrades were the key
A stable API is not going to change randomly on you, and of it does, it's going to come with deprecation policy so you have time to do it.
Personally I did have few "painful" k8s upgrades, but every time it happened it was due to long-announced large scale change... That another team didn't want to upgrade over despite delaying so long the last version compatible became EOL
Infrastructure is a high rise building. Long term planning and careful maintenance is needed. And it shouldn't be necessary to replace the foundation every year or two.
Two words: companion plantings
Also, there's a world of incompatible plants. Some plants actively harm the growth of others, and some simply can't grow in the same place because of different soil and nutrient requirements.
I feel like software engineers who preach that everything must be connected to the internet and update from the mothership regularly are fundamentally disconnected from reality. If your design is robust to begin with you should be able to depend on it without constantly fiddling with everything.
this LTS hysteria is completely made-up
it benefits from updates every year, also who uses naked k8s (the hard way?), folks use a distribution with an updater
of course it's more work than simply applying the new charts, but the nice thing about k8s is that you can dump out the stuff from the working one, use k3s/kind/minikube, try the upgrade, and you are as good to go as with a dist-upgrade or similar.
That being said, you build gardens with soil and highrises with concrete. Kubernetes is soil, not cement.
1. https://kubernetes.io/docs/reference/using-api/deprecation-p...
https://abseil.io/about/philosophy#we-recommend-that-you-cho...
However each upgrade of k8s needs planning. Need to check through the list to see what APIs are broken, and figure out if anything needs changing, preparing for it, including stuff the cloud provider has chucked in the mix.
Probably to the point I feel like a cluster of clusters would be safer, so you can slowly roll it out.
Plan regular upgrade so it's moved in small chunks and so that teams are aware of where they stand with APIs etc.
this sounds great in theory. after 5years of running a startup, you learn to pick your battles. that db library you upgraded? works great except for this one edge case where they changed the interface in an obscure portion that affects 5% of your users. it got past QA but thaat 5% of users get REAL VOCAL about it. now its in production and you're planning an emrgency revert of that dependncy after you verify that you aren't using any of the new features of the library.
sounds doable? now multiply that by the number of dependencies your app has.
I agree that it good to periodically update deps and allocate time for keeping your system up to date but Its easy to let tracking all your deps suck up all the time your startup would be better off spending on adding features that bring new users in and add to your bottom line.
When your rock solid dependency releases a new major version, when you’re ready you update it, and fix all the things you need to in a single PR, you take that to prod when you’re ready, and roll it back if it’s unhappy, or your favourite variation of out-and-back.
What I’m advocating against, is picking a version, and then sitting on it for so long that it inevitably goes out of support, and then find yourself having a really bad, un-fun time-probably at some awful time of day- and discovering the work you need to do to get out of the current system is so much more than you initially realised, because you’re so far behind.
Breaking changes happen anyway at some point, you may as well stay abreast of them.
In theory, I agree. Tend your garden, do small upgrades often instead of major upgrades less often.
In a homelab this is easy to implement. In a small organization it is too.
But in the facetious "real world", things are a lot more complicated. I work as an SRE Manager and my team is basically ALWAYS upgrading Kubernetes. New releases drop about as fast as we can upgrade the last ones.
When you work on a large cluster, doing an upgrade isn't a simple process. It requires a ton of testing, several steps, and being very slow and methodical to make sure it is all done properly. Where I currently work, we have 2 week sprints and infrastructure changes must align with sprint cycles. So to promote an upgrade at the fastest possible schedule it requires:
- Week 0: Upgrade Dev environment
- Week 2: Upgrade QA Environment
- Week 4: Upgrade Sandbox Environment
- Week 6: Upgrade Prod Environment
That is the fastest possible schedule. That assumes we do a cluster upgrade every sprint, which is 2 weeks. It also ignores other clusters for other business units. We have 4 primary product lines, so multiply all that work times 4. Plus we have supporting products (like self-hosted gitlab, codescene, and custom tools running in k8s clusters).
I say fastest possible schedule because, we can't keep up with this schedule, but even if we could it is the fastest we could go and still maintain our deployment and infrastructure promotion policies.
With new releases every 3-4 months (12-16 weeks), we are essentially in a constant state of always upgrading kubernetes. Right now my team is 2 versions behind. Skipping versions doesn't make sense because you can't guarantee a safe upgrade when skipping versions.
This is why LTS releases are nice. When you run systems at large scale, it is impractical to upgrade that often. I'd prefer to limit upgrades to no more than twice a year and personally I find annual upgrade cycles to be the best balance between "tending the garden" and "not drowning in upgrade work". LTS releases are usually tests to skip upgrades so that companies can go from LTS release to LTS release, without the need to worry about upgrading every minor version in sync.
Remember, upgrading K8s clusters isn't what my bosses want to hear my team spends our time. They want to know that observability is improving, devs are getting infrastructure support, we are building out new systems, deploying hardware for the product team, running our resiliency tests, etc.. Sure upgrading is part of the job, but i can't be ALWAYS upgrading. I have a lot of other responsibilities.
Hypothetically, since the scaling limit for Kubernetes clusters is ~10,000 nodes (last I checked), you could have multiple product lines that took up 10k+ nodes each. Then there's no reason why not to split by product line. But in the beginning it should be fine.
There's also edge cases - Kubernetes doesn't support setting resource requests or limits for networking or I/O, which you usually solve by setting up taints/tolerations/affinity to manually schedule those workloads onto nodes where you've manually run the numbers. But still not usually a reason to prefer separate production clusters.
Of course that's feasible, to tell the Kubernetes scheduler that you only want to schedule the workload on a very specific server, but that's not really best-practice...
Easier to manage because less dependencies : you are the sole user, so you can do whatever you want
This is the greatness of public cloud, where multi team synchronisation can be reduced a lot
For very small projects, one cluster per product is expensive, so mutualisation makes sense. Past a certain size, this is not the case
IME mixing financial concerns with engineering concerns puts you into an architectural corner. Suddenly the business wants an integration between the two projects - which cluster does the integration run on? Do you spin up a whole new cluster / account just for the integration? Inside a project, you need to run dedicated infrastructure for large customers. How do you report that for billing?
Just set up tagging correctly.
> For very small projects, one cluster per product is expensive, so mutualisation makes sense.
If you're not working for a BigCo, every project starts out very small ;) You shouldn't optimize prematurely.
About integration, I do not understand what you say You have a provider and a consumer, each of them build (and pay) their own parts
If service A is in one cluster, and service B is in another cluster, where billing is per-cluster, in which cluster do you put the ETL integration? It belongs to either both or neither, depending on how Finance wants to bill it.
a) if you're going to intentionally make the trade-off to have higher costs in exchange for reducing ops load and simplifying administration within an account, where services make network calls across accounts, then adopting ECS/Fargate is probably a significant improvement over Kubernetes.
b) The underlying engineering / financial reality is still there and very much a leaky abstraction that will show up on your AWS bill. Cross-VPC, cross-region, and cross-AZ networking costs are very real overhead that you must consider; you either still have centralized network planning with shared VPCs and subnets or you basically decide to give up and let AWS send you the bill when teams create their own VPCs, their own subnets, etc.
c) setting up DNS / service discovery and network controls within a single cluster is simple. Doing so across account boundaries is not, and deciding to set up solutions like AWS Transit Gateway incur their own costs.
I'm sure it simplifies some things for some people, but Finance is going to get upset pretty quickly. Once you start to go down that path, it's very difficult to back out.
I am slightly curious why an upgrade takes 2 weeks? At my current work, rolling out an upgrade (self managed clusters), rolling out an upgrade on the “happy path” is a fairly low-intensity task: we kick it off, nodes gracefully drain, upgrade and come back online. No intervention necessary. Unhappy path just requires another command to roll them back. Prod is the same, but happens slightly slower because there’s more nodes (and more independent prod clusters).
I’m also not sure of why you manage all these clusters, why not merging dev, qa and sandbox in a single cluster with namespace partitioning? It would be way less work and probably more cost effective
Make upgrades easy, automate the tedious parts, and do them as often as possible.
If you do upgrades once per several years, yes, it is going to be excruciating.
At the end of course the decision will be based on business and market analysis, as even the cloud native foundation abides more or less to the same rules as everyone else does... Introducing LTS for kubernetes however will be a huge step towards pushing it down the enterprise products alley, where software is selected more for the amount of people available in the market, capable to work with it / operate it, and the running costs generated, rather than satisfying an actual need of the business.
Chances are that if your org needs LTS for kubernetes, then kubernetes is not the right solution to your problems. Which is probably the case anyway... but that's a whole different story.
LTS doesn't mean it's immune to bugs or security vulnerabilities. It just means that the major release is updated and supported longer - but you still need to be able to apply patches and security fixes to that major release. Yes, it's easier to go from 1.20.1 to 1.20.5 than to 1.21, because there's less chance of breakage and less things that will change, but the process is pretty much the same - check for breaking changes, read changelogs, apply everything. The risk is less, might be slightly faster, but fundamentally, it's the same process. If the process is too heavy and takes you too long, having it be slightly faster won't be a gamechanger.
So LTS brings slight advantages to the operator, while adding potentially significant complexity to the developer (generally backporting fixes into years old versions isn't fun).
The specific proposed LTS falvour is also hardcore, without an upgrade path to the next LTS. The exact type of org that needs an LTS will be extremely reluctant to having to redo everything, 2 years later, with potentially drastic breaking changes making that change very hard.
If you're at a point where a patch for LTS looks like an upgrade to the new version you've screwed up LTS.
Also, getting to the point of having an LTS and actually providing the support is expensive. You need experts that can backport security fixes and know the product inside out.
How do you do that on something as complex and with as many moving parts as Kubernetes? And how do you as an operator update that many things without checking there's no breaking changes in the patch?
eh, the kernel is an incredibly mature project with 1 machine scope. The kernel also has decades of operating systems research and literature to build on. Kubernetes in comparison is new, distributed and exploring uncharted territory in terms of feature set and implementation. Sometimes bad decisions are made, and it's fair to not want to live with them forever.
The kernel project looks very different today than it did in 1999.
There is a happy medium though, especially that Kubernetes is kinda far from it.
Erlang and its runtime discovered and solved most of the problems in the 80s. We are slowly rediscovering this the same way react discovered the event loop that Windows had discovered in the 90s.
Nomad is a good solution.
If you are talking about upgrade from LTS to LTS, can you give an example project where that is effortless? And if so, how do they manage to innovate and modernize without ever breaking backwards compatibility?
LTS to LTS is another story. But the point is that L=LongTerm so in theory you're only going to do this exercise twice in a decade.
> manage to innovate and modernize without ever breaking backwards yeah. fuck backwards compatibility. that is for suckers. how about stopping the madness for a second and thinking about what you are building when you build it?
So I've seen things like this in corporations many times and it typically works like this...
Well trained team sets up environment. Over time team members leave and only less senior members remain. They are capable of patching the system and keeping it running. Eventually the number of staff even capable of patching the system diminishes. System reaches end of life and vendor demands upgrading. System falls out of security compliance and everything around it is an organizational exception in one way or another. Eventually at massive cost from outside contractors the system gets upgraded and the cycle begins all over again.
Not being able to upgrade these systems is about the lack of and loss of capable internal staff.
In 2022 they had to start a new huge effort do deal with migration off CentOS7, and the problems were so painful it was considered reasonable to build a Linux distro from scratch and remove all traces of distro dependency from the product (SaaS)
Alma and Rocky were considered, but that would still involve (possibly similarly painful) migration as with CentOS 6 -> 7.
Have you seen pricing for RHEL? We're talking hundreds thousands of systems. I never seen raw stats, but I would have been totally unsurprised to see them hit million instances across all clouds used, at least occassionally.
Decoupling software from distro dependencies was seen as a way to future proof deployment story and avoid situations like we had with CentOS 7, where they really, really would have liked upgrading some stuff for newer APIs, but couldn't due to mess with OS-provided dependencies.
The cost either way was likely millions and millions of dollars. But now they are having to do it all at once and risk breaking workflows for tens of thousands of people in a multitude of different ways.
I had to start by figuring all bits necessary to build not just kernel, but also external modules and attendant tools, using separate backported compiler because then-current LTS kernel wouldn't compile using distro-supplied GCC.
Unless you're relying on buggy behaviour, there should be no breaking changes in an LTS update.
(...of course, there's no guarantee that you're not relying on buggy (or, at least, accidental) behaviour. People relying on `memcpy(3)` working as expected when the ranges overlap, simply because it happened to do so with historic versions of the `libc` implementation they most commonly happened to test with, is one example. But see also obxkcd https://xkcd.com/1172/ and https://www.hyrumslaw.com/ )
Or a security vulnerability has forced a breaking change. Or any other issue, which is why you have to check.
Theoretically, I suppose?
Do you have a historic example in mind?
I've been running Debian "stable" in its various incarnations on servers for over a decade, and I can't remember any time any service on any installation I've run had such an issue. But my memory is pretty bad, so I might have missed one. (Or even a dozen!) But I have `unattented-upgrades` installed on all my live servers right now, and don't lose a wink of sleep over it.
Yes, it's Ubuntu, but doesn't matter - sometimes security fixes require a breaking change and there's nothing that can be done to avoid it.
The worst one I know: for a while basically all Cloud Foundry installations were stuck behind a patch release because the routing component upgraded their Go version and that Go version included an allegedly non-breaking-change that caused it to reject requests with certain kinds of malformed headers.
The Spring example app has a header with the specific problem impacted. And the vast majority of Cloud Foundry apps are Spring apps, many of which got started by copying the Spring example app.
So upgrading CF past this patch release required a code change to the apps running on the platform. Which the people running Cloud Foundry generally can’t get — there’s usually a team of like 12 people running them and then 1000s of app devs.
For the things I was doing on the hooks, it became clear that I needed to make changes and get them added upstream, rather than doing it in hooks, but that meant we were running OpenSSL with local patches in the interim of upstream accepting and releasing my changes. If you're not willing to run a locally patched security critical dependency, it puts you between a rock and a hard place.
The model seems to largely be, the CNCF/Kubernetes authors have done a good job of writing clear expectations for the lifetime for their releases. But there are customers who for various reasons want extended support windows.
This doesn't prevent the distribution from offering or selling extended support windows, so the customers of those distributions can put the pressure on those distribution authors. This is something we offered as a reason to use our distribution, that we can backport security fixes or other significant fixes to older versions of kubernetes. This was especially prevalent for the customers we focussed on, which were lots of clusters installed in places without remote access.
This created a lot of work for us though, as whenever a big security announcement came out, I'd need to triage on whether we needed a backport. Even our extended support windows were in tension with customers, who wanted even longer windows, or would open support cases on releases out of support for more than a year.
So I think the question really should be, should LTS be left to the distributions, many of which will select not to offer longer releases than upstream, but allow for some more commercial or narrow offerings where it's important enough to a customer to pay for it. Or whether it should be the responsibility of the Kubernetes authors and in that case what do you give up in project velocity with more work to do on offering and supporting LTS.
I personally resonate with the argument that this can be left with the distributors, and if it's important enough customers to seek out, they can pay for it through their selected distribution, or switching distributions.
But many customers lose out, because they're selecting distributions that don't offer this service, because it is time consuming and difficult to do.
Is it normal for a distribution to also package upstream projects that customers want?
It's a great point which probably should be part of the discussion. Say even if Kubernetes project offered LTS, how would that play into every other project that is pulled together.
> Is it normal for a distribution to also package upstream projects that customers want?
I suspect it differs by distribution. The distribution I worked on included a bunch of other projects, but it was also pretty niche.
Exactly this. I see a lot of parallels between k8s releases and OS releases. Even if you're paying microsoft for patches to windows XP, I'm not seeing any of that and the python runtime that most of my software relies on also isn't seeing their cut so... I guess upgrade to at least python 3.10 and then call me back?
I would prefer to see the conversation turn more to "what can be done to reduce reluctance to upgrading? How can we make k8s upgrades painless so there's minimal incentive to stick with a long out of date release?"
Sure, but if they really need that service they will gravitate to distributions that do provide it, so, I think, no harm done here. It's to me like JDK distributions. Some give you six month, some give free LTS and others give you LTS with a support contract. LTS with backports is work, someone has to pay for it, so let those who really need it pay. Everyone else can enjoy the new features.
tl;dr: I'm with you in the camp that you can leave it to the distributors.
That should be Red Hat's job, just like they do with RHEL.
In theory, there's nothing stopping you from just updating the kubelet binary on every node. It will generally inherit the existing pods. Nomad even supports this[1]. But apparently there are no guarantees about this working between versions. And in fact some past upgrades have broken the way kubelet stores its own state, preventing this trick.
All I ask is for this informal trick to be formalized in the e2e tests. I'd write a KEP but I'm too busy draining nodes!
Anyway, make upgrades less scary and more routine and the risk bubbles away.
This sounds a lot like "We don't actually patch the OS" which is quite common among many companies.
As a former enterprise kubernetes distro maintainer, I can tell you with certainty that most on-premise kubernetes customers aren't patching their machines between kubernetes releases, and try to stay on kubernetes releases for 18+ months.
I am pretty sure Kubernetes itself does not mandate node draining. I have been doing upgrade for bare metal cluster for years, and like you said, it's mostly just replacing kubelet binary and bounce.
However, I do understand in public cloud it's usually recommended to perform a node rolling update instead of modifying online nodes in place. Actually, I prefer this way because of the benefits of immutable infrastructure. The downtime is unfortunately, but so far I have been enjoying working with devs to better designing reliable apps to tolerate node issues like this.
The procedure is more of a cloud-ism where people don't upgrade their nodes in place but rather get entirely new nodes.
Additionally, I think one of the major reason for LTS is that K8s (and related software) regularly introduces breaking changes. Out of all the software that we use at work, K8s probably takes the most development time to upgrade.
They are probably at extent of what they can handle with 2 years without Kubernetes Team getting onboard.
I'd like a slightly longer LTS purely so I'm not having to spend all my time spinning the plates to keep things up. I don't need 10 years LTS, I need three so I can work with the rest of the enterprise that moves even slower.
That is: an initial high failure rate (teething problems), a low failure rate for most of the lifespan (when it's actively maintained), then gradually increasing failure rate (in hardware this is called wear-out).
Unlike hardware, software doesn't wear out but the interfaces gradually shift and become obsolete. It's a kind of "gradually increasing risk of fatal incompatibility". Something like that.
I wonder if anyone has done large-scale analysis of this type. Could maybe count CVEs, but that's just one type of failure.
In my experience:
- AKS is the simplest to update: select "update cluster and nodes", click ok, wait ~15m (though I will always remember vividly the health probe path change for LBs in 1.24 - perhaps a giant red banner would have been a good idea in this case)
- EKS requires you to manually perform all the steps AKS does for you, but it's still reasonably easy
- All of this can be easily scripted
I totally agree with the other comments here: LTS releases would doom the project to support >10y-old releases just because managers want to "create value", but don't want to spend a couple weeks a year to care for the stuff they use in production. Having reasonably up-to-date, maintainable infrastructure IS value to the business.
If you can't keep up you have two options:
1. Pay someone else to do it for you (effectively an LTS)
2. Don't use it
Software is imperfect, processes are imperfect. An LTS doesn't fix that, it just pushes problems forward. If you are in a situation where you need a frozen software product, Kubernetes simply doesn't fit the use case and that's okay.
I suppose it's pretty much all about expectations and managing those instead of trying to hide mis-matches, bad choices and ineptitude. (most LTS use cases) It's essentially x509 certificate management all over again; if you can't do it right automatically, that's not the certificate lifetime's fault, it's the implementors fault.
As for option 1: that can take many shapes, including abstracting away K8S entirely, replacing entire clusters instead of 'upgrading' them, or having someone do the actual manual upgrade. But in a world with control loops and automated reconciliation, adding a manual process seems a bit like missing the forest for the trees. I for one have not seen a successful use of K8S where it was treated like an application that you periodically manually patch. Not because it's not possible to do, but because it's a symptom of a certain company culture.
But: most of that can be mitigated by keeping a common structure and baseline templates. You only need to validate your common structure against a QA cluster and then roll out necessary changes onto the production cluster... but most organizations don't bother and let every team roll their own k8s, pipelines and whatnot. This will lead to tons of issues inevitably.
Asking for a Kubernetes LTS is in many cases just papering over organizational deficiencies.
AWS released LTS version of EKS last month. 1.23 is on Extended support until next year for free. But later versions will cost money.
https://aws.amazon.com/blogs/containers/amazon-eks-extended-...
I'd rather not see the resources in the Kubernetes project being re-directed to users who are in a situation where they aren't able to do a well-known action at planned intervals two or three times per year.
Two of the reasons we gave an ultimatum to teams to get off K8s is that CNCF doesn’t like LTS and loves introducing breaking changes about every other version.
Unless you have a third party product with a helm chart that really needs to run on EKS, our dev teams have been told to use ECS
We don't allow systems that cannot be mass-managed and don't have an active lifecycle that keeps up with demand. Sometimes that doesn't mean K8S, but often it does because it's the only large adopted system with a high speed reconciler. But the teams get to choose that on their own.
Edit: let me rephrase that a little.
It seems that software lifecycle speed and business speed interact to a degree where if the software is faster than the business, you need a different product or speed up the business. For me, that would mean that if either aren't an option, I'd change jobs because it doesn't align with my standards.
Only recently they released paid extended support in preview, which extends support for an additional year.
You can't get a really old k8s version on gke
> This bot triages issues according to the following rules: ...
> /close
As a business, you can decide to become more efficient, or you can decide to try to support the status quo that enjoys internal political equilibrium. Smart businesses go for the former, most businesses go for the latter.
I don't think K8S should create an LTS, I think they should make it dirt simple to update.
If you want to migrate from 1.24 to 1.28, you need to upgrade to 1.25, then 1.26, then 1.27, and only then can you go to 1.28. This alone is a significant impediment to the normal way a project upgrades (lag behind, then jump to latest), and would need to be fixed before any discussion of an actual LTS process.
I guess too many people are using k8s where they should have used something simpler. It's fashionable to follow the "best practices" of FAANGs, but I'm not sure that's healthy for vast majority of other companies, which are simply not on the same scale and don't have armies of engineers (guardians of the holy "Platform")
IMHO , infrastructure is too low level for frequent updates. Older versions need LTS.
at the end of the day it’s about cramming a ton of functionality into something like YAML is it not?
But not having LTS is really difficult. It is impossible to keep rewriting things. And all related components: terraform, helm, GKE etc also update too quickly for us to manage with a small team. We do some lower level infrastructure work so these problems bites us harder.
- pray to god whatever helm chart dependencies you have didn't break horribly, and if they did, that there's a patch available
Unless you're saying that the CRD creates generic kubernetes resources in the cluster that is owned by the CR. In which case, the tool should definitely pick up the generic object at least, but not necessarily tell you that you need to update your version of the CRD, again because the CRD is owned by a third party.
Given that the kubernetes dev cycle is so fast and API changes are so frequent, combined with the fact that the kubernetes API is designed to be built on top of, this tends to be a real hassle in the real world was my only point.
Compare it to something like the “old” way of doing it, some monolith running on a container in a VM - that stuff could run until the cows come home with very few issues or intervention.
I don't think this is a problem that is unique to kubernetes, is what I'm getting at. You never said that it was, I'm mostly thinking out loud.
May be kubernetes is too young at this point...
Survey at https://bit.ly/k8s-upgrade-survey
After all, we could also looks for a discussion in the article about the need for an STS, and how that should defined before GA or after the RC. Definitely at least before the EA versions of Kubernetes and at least before EOL of the product....
The actual issue that needs to be addressed when upgrading is API deprecations and incompatibility. For better or worse, when people try to analogize this to the way Red Hat or Debian works, Kubernetes is not Linux. It may not always perfectly achieve the goal, but Linux development operates on the strict rule that no changes can ever break userspace. Kubernetes, in contrast, flat out tells you that generally available APIs will stay available, but if you use beta or alpha APIs, those may or may not exist in future releases. Use them at your own peril. In practice, this has meant virtually every cluster in existence and virtually every cluster management tool created by third parties, uses at least beta APIs. Nobody heeded the warning. So upgrades break stuff all the time.
Should they have not ever released beta APIs at all? Maybe. They actually did start turning them off by default a few releases back, but the cultural norms are already there now. Everybody just turns them back on so Prometheus and cert-manager and Istio and what not will continue working.
You need to keep updating it!
No legacy 10 year old garbage in the corner just because managers don't understand that this is shit.
And for me it's the perfect excuse NOT having those garbage discussions.
And honestly k8s doesn't change that much anyway
But there is no equivalent system for whole clusters. To transition from one cluster to another, you have to handle everything that k8s gives you without relying on k8s for it (restarting services, rerouting traffic, persistent storage etc). If you can do all that without k8s, why use k8s in the first place?
So no, in practice clusters, just like any other stateful system, are not really cattle, they are pets you care for and carefully manage. Individual cluster nodes can come and go, but the whole cluster is pretty sacred.
But my preferred way is in place as it's easy: upgrade one Ctrl plane node after the other, than upgrade every node or create a new one and move your workload.
The power of k8s is of course shit when you only move legacy garbage into the cloud as those are not ha at all.
But lucky enough people slowly learn how to write apps cloud native.
Upgrading one control plane node after another can still break your cluster when you have your control plane nodes running incompatible versions.
If you get a big worker node failover during the control plane upgrade, which of the control plane nodes should provision the new worker nodes, which versions of containers should kubelet pull, etc?
The upgrade process for a running multi-node solution is an extremely difficult technical challenge, and it has to be designed very carefully to actually have a good chance of working. Breaking changes are almost impossible to introduce if the goal is to upgrade gracefully - you probably need a special compatible version to run while the upgrade is still in progress, that is neither the old version nor the targeted new version, but can handle both. You need to make sure all APIs are versioned in a way where it's easy to distinguish a request coming from an old client (so you can serve the old format) or a new one etc.
Kubernetes has done some work on this, but it still often requires manual work. And many common plugins have just not put in this work and will break if you try to upgrade a live cluster.
I'm upgrading k8s for years now, had very rarely issues.
Independent of it if you really have big workloads it's the right thing anyway to have a good team behind it.
Nonetheless if you use gke for example I and no complicated plugins, it's basically automatic.
Ah, but there are. Maybe not a cluster orchestrator that would let you hot swap a single cluster, but if you're only running one cluster it's inherently a pet already.
One common way that I've personally used to solve this problem is use Federated Clusters (kubefed) to run multiple clusters with global load balancing so that even when one entire cluster goes away its traffic is just shifted to the other cluster (whether because of upgrades or unexpected outages of the control plane)
So overall, you need all, or most, of the features of kubernetes in your cluster federation solution. So why then not make each of these individual kubernetes clusters really small, say a single-node cluster each, and rely on the global infra to manage between them? Hey-presto, your "cattle farm" is now a single big pet.
Keep 3-4 clusters. Deploy your new application with new addons in new cluster. Keep old application running on old ones.
When people run a rolling release on their server, their original intent is "Yay I'll force myself to be up-to-date". Reality is, they get conflicts on installed 3rd party software in each upgrade. What ends up happening is, they get frozen on some point in time, without even security patches for a long time.
k8s is like an OS, it's not just the core components, there is <ingress/ gateway/ mesh/ overlays/ operators/ admission controllers/ 3rd party integrations like vault, autoscalers, etc>. Something or the other breaks with each rolling release. I've grown really tired of the way GKE v1.25 pretends to be a "minor" automated upgrade, when it removes or changes god knows how many APIs.
This is what is happening in kubernetes land. The broken upgrade fatigue is real, but it's hampered by wishful thinking.
Azure are the ones who effectively posed the question in the community of hey, since we all appear to be doing this is there any value in collaborating on it in the open by rebooting the LTS working group.
Now that said, while you are absolutely right that third party solutions are the biggest boat anchor to users upgrading whether commercial or open source, they are also the reason an LTS doesn't necessarily fix much because then those things need an LTS too and/or they decide hey this is great we will fix on this one and it becomes an even bigger lift when the ISV does need to bring their stuff up to latest.
I agree with comments in this thread that LTS should be a commercial bought option only.