On the future of free long term support for Linux distributions
utcc.utoronto.ca
utcc.utoronto.ca
One of the things that's quite popular
with people out in the world is being
able to set up a Linux server and then
leave it be for the better part of a
decade without having to reinstall it
or upgrade the distribution.
I have never seen anybody who runs a business being happy about something new in the next version of the distro which powers their onlineshop or blog or whatever they run. But I constantly see how annoying it is for real world applications having to update the "outdated" stack every few years.
The stack is almost always already perfectly fine for what they are doing. All it needs is security updates. Which get less and less frequent as the system matures.
There should be an effort to offer 50 years of security updates for Debian 12. That would create more value for real businesses than all the work that will go into Debian 13, 14, 15 etc.
So I suggest it is worth building the next version 5% slower and use the free 5% of resources to fix security issues in the last one.
And by this, extending security fixes for the last version to 50 years.
People will expect the distribution to just work with newer hardware. Or maybe you're saying upgrade to debian 13 if/when you get new hardware but stay with 12 for fifty years on current hardware?
Intel and AMD should fire their marketing if a lot of people feel ok running the same hardware for fifty years, no?
Why would you need support for new hardware if the use case is hardware that's going to sit for 50 years doing one thing and doing it well?
2. Is making a new release really more work? Most parts of a distro come from upstream, so their development too. I don't think most tools have stable branches, you're just expected to upgrade. Backporting security fixes is then actually more work than simply updating and solving breakage every now and then.
The alternative is to maintain your own patches, but that is not remotely sustainable for even the largest commercially supported distributions.
I sidestep all these concerns by running a rolling release distro on my workstation and deploying serverless code at work. I don’t miss the days of trying to get software downloaded off the web to work on a fixed debian or Ubuntu install.
Edit: nor having my production language runtime constrained by distro!
I wholeheartedly agree with the sentiment, but it sounds like a wishful thinking. If you maintain a simple library, do you backport a bugfix to every single previous released version? Because that's what would be required. From every single one of them.
If we want to bring math into the conversation, then I don't think that would cut it, since supporting old releases is basically quadratic in terms of human effort. It's why distros ship LTS releases at a different cadence, which makes the problem less bad. But the costliness explodes if they do it for any longer than a few releases. It'd be fine if all the sticks in the mud could agree to use one specific old version, like RHEL5 and only RHEL5. But that's basically what projects like the BSDs are already doing, and they're awesome. I wish people who desire Linux to not change so rapidly were supporting the BSD folks instead, because they actually live up to preserving UNIX in a very authentic form.
50 years of support doesn't make sense for these usecases, as the world keeps moving on around them. A blog that still looks like it's 2008 won't sell your business in 2023. A webshop from 2013 won't accept the popular payment methods of 2023 in large parts of the world. If these products will be updated anyway, at a certain point building on an old stack causes more work than it saves.
They are not for selling or self-promotion using the newest shit. These sites are for getting a job done. And users actually enjoy it very much when the http form they use daily stays the same for over a decade.
In 50 years, we'll still want a forum-like to discuss about the latest shenanigans of then-Elon Musk.
blogs can be personal, or something of historical interest
That reminds me, SQLite is engineered to keep working until atleast 2050 (and beyond).
You would have need to adapt your code along the way but if you wrote a C++ application or service 30 years ago, there is a good chance of making that run on modern Windows as well.
I'm not a Windows, nor a C++ guy, but that seems like it would have been a pretty safe choice if you built an application back then with the expectation of having it running 50 years later.
Windows is a really interesting stark contrast to the rest of the computing world that prefers moving fast and breaking things, much to the chagrin of most users.
Probably many of Burroughs B5000 applications will still mostly run without any issues on a recent Unisys ClearPath MCP, just as one possible example.
https://vmssoftware.com <-- seems to be the company doing VMS these days
The benefit of open source software is that as long as the code is out there, somebody could take it upon themselves to fix things and share the results. You don't have that option with windows. And you only have that option with Dos because somebody created an OSS version of that; which is still being used by some to run stuff from last century. Windows is probably a bit more tricky.
That's caused by a "temporary" fix in an underlying library, which seems was not so temporary, given that it got into the feature freeze and release of the Ubuntu LTS I use.
This means that for the whole support period of any distro that bundled the broken feature, users will find it and have to work around it, over and over.
Can you imagine if that was for a century?
I think distros should relax their policies and not only accept changes to fix security vulnerabilities (as is the case now), but also to fix promised features that shipped in a broken state. Otherwise, once some broken software ships, it stays broken for the whole lifecycle of the distro, which is bonkers.
[1]: In case anyone can provide further feedback, ideas for the author on how to fix it, or any help. https://gitlab.gnome.org/GNOME/vte/-/issues/2504
If i had to keep a software stack for a century, i wouldn't pick GNOME. Generally you'd avoid any software project that has the faintest smell of "move fast and break things".
-- ex GNOME user
But again, people have different conceptions of lagging behind. Some people think not having a graphical installer is to "lag behind", when this is pretty much a design decision and not a problem of fewer people working on new things.
[1] https://www.openbsd.org/innovations.html [2] https://undeadly.org/cgi?action=article;sid=20230714121907
Linux has had a lot of work done to support complex features and scaling in a number of aspects, whereas OpenBSD has not. In fact, the insistence on security and keeping things conceptually simple, almost inevitably means they pass on some significant improvements. When you want pure power, OpenBSD is just not the best choice, for example; and the virtualization stack is very primitive.
That said, the upgrade process (like most things) is thoroughly documented every release, and a given release will continue to chug along for years, so I can see where you're coming from there.
Fairly slow incremental changes over time, and no real surprises on a new release.
It's incredibly boring, but in the right way.
Gnome has adapted to changes in computing form factors, supporting more mobile setups (touch and tablet) that emerged over the past 15 years while other DEs may mature around a Desktop/Laptop setup.
- mostly happy Gnome user
It's less sad than thinking that they do it to themselves.
The decade of (security) fixes (free or otherwise) smells like neglect. This is probably fine with 80s or even 90s software. But for internet connected devices I think it's bonkers to expect others to maintain and support the foundation of your software.
That is, assuming the software is functionally done, finished and no longer maintained. If whatever you meant to build, is still maintained and actively supported, it should not be too much to expect the OS to evolve with it. Upgrade to the latest platform every now and then.
Yes, migrations can be painful, but grow up: test thoroughly and get your ducks in a row. If that's too much to ask, you're simply lifting beyond your weight.
This ties in very well with a discussion that happened a couple days ago about whether the concept of "finished software" even exists:
There's the problem: developers primarily think of themselves, expecting end users to be power users not unlike developers.
That doesn't hold. Developers can handle change & a rapid upgrade cycle. End users... not so much. Or not at all.
The only updates end users care for are risk-free, invisible, hassle-free (as in: automatic) updates that plug security holes, correct broken functionality, or minor feature updates that don't interfere with anything already there. For example drivers that support new hardware which uses existing system bus w/ existing software infrastructure.
Update can't fit into this frame? Then put it in next, feature-breaking release. Not in currently-stable, supported one.
Imho, Linux currently has a pressing need for community maintained distro with 10y+ LTS cycle. Or existing distro that offers 10y+ support on selected releases. Such distro could easily send RHEL into obscurity.
However there were plenty of severe flaws that couldn't be fixed without some fundamental changes.
What people appear to want is all the problems fixed, but none of the hassle of actually having anything change to accommodate these fixes.
fixing bugs sucks. That's why rewrite the whole thing.
Your 20 year old debian won't be able to provide a TLS stack that is compatible with any modern browsers. And you could argue that this is still a "security backport", but backporting a new TLS version isn't like your garden-variety buffer overflow backport patch.
HTTP + merkle-tree based integrity checking + having the merkle root in DNS could be way more stable. Even the the SHA1 deprecation would barely affect this, as long as you're not using the hash tree to vouch for content provided by 3rd parties collision attacks don't matter.
I wasn't assuming you would. That's one of the things that have to go on the chopping block if long-term stability is the goal.
Do you genuinely think people want computers that cannot be used to read email, do online banking, buy things with, watch Netflix, and so on?
> trust their local resolver
No thanks. There was a time when software would trust devices on the LAN. I'm not going back to the dark ages
The protocol has the security bug. The protocol needs to be changed to fix the security bug. The old stuff stops being able to talk with new stuff.
That's the problem poster above was talking about.
Cert authorities are relatively much simpler problem to solve as it is just few files to update
Real businesses seem to prefer to spend time dealing with breakages than money to avoid them. It's not economically optimal.
Some buy their OS from IBM instead with mixed results but generally more focus on doing things businesses seem to want.
Nothing about computing is anywhere close to settled enough for anyone to actually think about it on century-long time scales. Maybe we're at a point where we can talk about appliance style computers-single task, very limited I/O lasting that long.
It'd be nice, don't get me wrong, but OpenBSD should probably last a century before we start asking if we could let it run for another century.
Some things are pretty well settled. 40 year old Unix software (K&R C,IPC/network sockets) would work on a modern linux with trivial or zero changes, and it doesn't look like it's gonna change soon. That was the age of not room sized, but half a fridge sized "minicomputers". Old software that people rely on doesn't necessarily need new features or improved capabilities that new standards and interfaces provide. Security, the constant change of the environment(network, protocols), and hardware and software changing in tandem are the only reasons people have to migrate or rewrite some pieces of well established software. Some of these changes in the environment feel as if these changes are for the changes' sake, to create programmer jobs.
Is that true? Will K&R C (that is, C with old-style function declarations, not prototypes) compile on a modern compiler? I thought that gcc, for instance, no longer supported that declaration style. (And if you just have to add a flag to the makefile, I will agree that it's a trivial change.)
On the sockets, I agree with you.
It's g++ that doesn't support this, gcc should.
Code written in COBOL on mainframes 51 years ago still runs today.
Ditto for Java 8.
I'm a developer and if I would be left to my own devices I would upgrade to the newest beta software where possible and run Java 21 EA with enable-preview.
The only stopper is lazy people and infosec.
The only thing that matters in large + long-lived engineering organizations is organizational buy-in, and that's only remotely possible with LTS, because anything with shorter upgrade cycles takes too much time to get too many other people to buy in before a new version is released (and the old version is no longer supported).
There's a reason why you see a ton of stuff like VBA in the military, and it's not because VBA is usually the best tool for the job by any technical metric. It's because the people who chose it didn't need to align 100+ other stakeholders in order to pick a different tool; they were able to start implementing what they needed immediately.
This is an archaic view and certainly not true for all Fortune 100 teams.
Plenty of them have digital platforms that bring in lots of money, and staying with legacy tech isn’t going to be good for business when faced with new, nimble challengers. Learning from the wider industry, FAANG included, is the smart thing to do.
> LTS
The key here is to save $$$ and stay secure. LTS is just deferred cost and pain.
For teams with good tests and CI for instance — if they’ve migrated to Java 11, upgrading the JVM is relatively easy after that.
Staying on LTS releases for ages is costly as you’re just deferring cost, as even Oracle JDK contributors have pointed out.
Ultimately it’s a cost/benefit equation, but often it’s economically sensible to bite the bullet and upgrade from Java 8 -> 11, and then upgrade more regularly.
Incidentally, Oracle’s recommendation is to perform Java LTS upgrades every 3 years. Sure you can defer further, but it’ll cost you. Oracle support ain’t free. But most well-run teams with tests and CI can move every 12-24 months. [1]
[1] https://www.reddit.com/r/java/comments/c5pl1q/adoptopenjdk_i... , https://www.reddit.com/r/java/comments/c5pl1q/adoptopenjdk_i...
Well-run by what criteria? Pournelle's Iron Law of Bureaucracy points out that any large-enough organization (certainly F100's are large enough to count) will be run by people who take care of the organization, rather than people who pursue the organization's goals.
Most such teams are "well run" if they provide people with stability. No surprises, no forced upgrades, no rocking the boat, just an environment that let's people clock in and clock out with a feeling that they work in a supportive environment that appreciates them. Too many companies interpret CI as something that prevents them from delivering their work, or something that forces them to stop what they're doing to fix the main branch. In these companies, throwing something over the wall to someone else is a feature, not a bug. And if the person on the other end of that wall's only request is to still keep shipping Java 8? Well fine, so be it.
I think now it is 2 years, because that's the span between LTS releases from Oracle.
Why are small businesses like that running their own servers? Pay for the most managed service you can - if it's an online shop, then something like Shopify; if it's a blog, then something like Ghost.
Running Internet servers isn't just about security updates, it's also DDoS protection, backups, and VM replacement when the underlying hardware fails, at the very least. Why would any business devote any headspace to that unless they're forced to in order to make a profit?
Because the more managed a service, the less of a commodity it tends to be and leaves you at the mercy of the service provider wrt price changes, future availability, general quality.
When I buy a car, I may own the car, but I'm at the mercy of whoever for parts and service, which becomes more and more true over time as the OEMs shove more and more computerized parts into cars and stop publishing service manuals. Does that mean that I should build my own car from spare parts, so that I can be free of dependency on any one upstream manufacturer? Of course not - the costs of self-maintenance are too high. Now, if I run the New York Police Department vehicular fleet, and I'm responsible for thousands of cars that can't afford to be off the streets for too long, with the budget for a bunch of mechanic salaries and all the latest tools, then sure, maybe that's a serious argument. But not if I'm a small business and I have one server running a simple application.
> your old links will stop working
As long as your main website hostname is the same, you have 404s set up such that you land on a page that links you back to the main landing page... nobody really cares.
> order histories will be lost
For online shops, invoices anyway need to be exported to accounting software so that taxes can be filed correctly. I seriously doubt that a transition like this causes order history to truly be "lost".
Look, obviously switching managed providers is rarely a seamless experience... but neither is it impossible or even necessarily fundamentally difficult. Indeed, it's not unheard of for managed providers to help people make transitions by offering import options... https://www.shopify.com/migrate Shopify have guides to help with migrating, as does Ghost: https://ghost.org/docs/migration/
It depends on the size and funding of your organization.
For larger organizations with plenty of funding, it is easier and cheaper to just pay someone to run your servers for you. There will be monthly bills, but "it's operating costs, not capital costs" so no worries.
For smaller organizations with less funds, it is cheaper to buy a server, put a LTS OS on it, and self-manage it. If your IT budget is inconsistent, then monthly costs are too much and do not make sense.
Well business is free to sponsor such effort, I'm sure Debian project will be happy with extra money. Just that cost will be steadily growing year by year because backporting new fixes to ancient code gets more and more expensive in general.
Also majority of security problems will not be "distro machine is running" but "app that is running on that distro". A container with app that can just run on modern distro is far more preferable method to support something for long time.
You can run legacy 24-bit mainframe apps on modern 64-bit mainframes fine.
You can use legacy 3270 "green-screen" CICS mainframe apps over TCP/IP using CICS Transaction Gateway, with basically no changes to your CICS app.
It's just that mainframe customers pay a lot for this convenience and stability.
I won't be around to validate this, however I am asserting GNU/Linux will not survive after everyone that was part of its genesis is long gone.
Something else will take over it, will still somehow support POSIX in some form, just like most UNIX workloads were eventually taken over by GNU/Linux
I'm 100% with you on this. I also think that if Linus were to die today, the kernel would fall apart within 10 years.
No formalized succession plan whatsoever.
If Linus Torvalds Got Hit By a Bus Would Linux Die? (2012) <https://www.serverwatch.com/server-news/if-linus-torvalds-go...>
If I get hit by a bus, Linux will go on just fine says Linus Torvalds (2015) <https://www.theregister.com/2015/06/17/now_i_can_die_happy_w...>
So what we get is bitrot, plain and simple.
The obvious alternative is to rent software. But now we have the offering party who doesn't want to pay for maintenance, either and, again, we end up with bitrot.
What we'd need is a way to visualize bitrot and have everyone regularly pay for a renovation.
Backporting fix are expensive, so we live on edge without testing.
You do need to apply updates, but that's the point of support. You can take any version of linux as is and run it for 20 years with no updates, no changes, and no support. In my experience, SUSE also rolls from major release to major release with very little problem, so upgrading to SLES 16 is probably an in-place upgrade.
EDIT: also, RHEL never did this right in my opinion. They lock the kernel for a decade, SLES updates the kernel.
Only if you ignore the whole picture and focus strictly on the workflow "find bug, get fix, patch old kernel, release".
Google gives Android, Chrome and other stuff "for free" because they are getting something else in another part of the chain.
I'm aware of the long public good history of Red Hat, so I say the above with a measure of respect.
I don't see this working out long-term for Red Hat. The "free loaders" will have to gravitate to another rigorously tested and long-term supported setup. It's unlikely to be RHEL because of the bad taste (i.e it won't force people onto RHEL). So someone else will step up (My guess either Rocky or Alma, next time Red Hat changes the rules).
I also can't for the life of me work out how this gels with the GPL. Correct my understanding -- I thought as long as it was licensed GPL if you changed it you had to provide the source code with it (etc etc). So that means if you backport a upstream contributed patch to an older version of RHEL you're bound to release said fix under the GPL.
Actually enforcing the GPL, that's an entirely different issue. Some jurisdictions have found that only the original authors of a line of code have standing to file suit under GPL, not end users.
The VMware case also demonstrated that even if you have an original author, at least in Germany, the bar is very high to prove that the work is infringed, basically to the point of impossibility.
I don't believe any court has found in favor of the GPL in regards to end user being harmed.
Edit: I would argue that it also does benefit Red Hat in second-order effects. An increased community using Alma/Rocky (and previously CentOS) means more finding bugs and fixing bugs. I know at least two companies that would still be on Debian if not for CentOS circa 2010-2011.
I'm out of touch with them so I have no idea where they went after CentOS stopped being CentOS, but I doubt it was RHEL.
It is also very easy, to take Debian 12, install Docker or Podman on it and just put everything into containers, that way the only thing that you need to work after an upgrade is your container runtime and you are all set.
If you are able to run multiple applications on one host, you are likely doing the things that containers do for you.