Balancing the needs around the CentOS platform
blog.centos.org
blog.centos.org
But then I actually tried out Stream, and thought deeper about it. I don’t really need to exactly match RHEL’s point releases. I don’t need 100% binary compatibility with RHEL. My installs don’t break between point releases or errata updates, why would a rolling release change that? I just need a rock-solid OS to run my applications on, and Stream seems to fill that need. Just ‘cause the update model was changed doesn’t mean the OS is unstable.
I agree with the authors claim that 95% of CentOS users will be perfectly happy with Stream. Of course, there is that remaining 5% gap, where I understand the frustration of those in that situation.
I already roll several other Red Hat upstream projects in production, including AWX (Ansible Tower) and oVirt (RHV). I’m sure RH would love for me to pay for support, but I don’t really need it. The upstreams serve me well enough.
The one thing I can’t get over though, is the decision to roll out this change in the middle of CentOS 8’s lifecycle, trimming out what, 8 years or so of support. That seriously stung. This decision should have been made a year or two ago, before CentOS 8 was released.
Literally every enterprise user I work with that has CentOS uses it specifically because it directly tracks RHEL. They don't want to pay for RHEL licenses in test/dev but need a binary compatible version to ensure they can get enterprise support in production should they hit an issue. Having different drivers/kernels/libraries is a non-starter.
There's absolutely no way that Redhat is so blind to their user base that they don't know this is happening. This is a pure money grab, if they can get even 5% of the users doing what I've seen paying for licenses, it's a net-win for them.
I'm 100% onboard with Redhat doing things to make lives difficult for companies like Oracle that are trying to make money off their hard work. I have no time for user-hostile moves like this, every justification they've given is hollow IMO which is why they're still trying to do damage control and persuade the public that it's not what it looks like.
Those types of users will leave CentOS and never look back.
RHEL is a subscription mess when working across its zoo ("Is our dev mirror the Azure's managed RHEL repository with mismatching kernel headers? Or pass through to RHEL's with epol disabled? Or RHEL 8 with docker repos disconnected so they can push their podman alternative?"). Losing centos hurts. As soon as you get into basics like drivers or build tools, RHEL quickly goes off the rails, and centos gave a semi-predictable escape hatch.
GPU land is directly on this path, so I feel awful for enterprise IT admins we work with who just want to do a good job for their client hospitals, schools, businesses, etc. We help shield them most of the way by reducing the install to just a gpu driver + docker, but those deps still need basic pinned version ranges. RHEL makes getting those hard, and worse, as life happens, make it hard to get the right diagnostic tool versions (gcc 7, ...). It's a bad look for a 2020 infra vendor to charge an enterprise for reliable and predictable rollout sw/support when RHEL makes it a marathon to get the right major release of base layers like docker/gcc/drivers. Contrast this to our cloud users, who one-click launch, or our self-managed cloud/ubuntu/etc. ones, who can generally get the right drivers + docker stack up in 10-15 provided commands.
Watching the chaos spread into centos makes me even more nervous about critical software relying on RHEL, and we already had to have The Talk with enterprises wanting HPC/ML on RHEL before this..
> Literally every enterprise user I work with that has CentOS uses it specifically because it directly tracks RHEL. They don't want to pay for RHEL licenses in test/dev but need a binary compatible version to ensure they can get enterprise support in production should they hit an issue. Having different drivers/kernels/libraries is a non-starter.
Unless you are doing kernelly things then the small difference in kernel version will not be that big a deal. Also for most applications, dev/test is better on Stream since it represent RHEL next. If the next version of RHEL will break your app, you find out in dev/test instead of in production like you used to. I think it's worth considering if this isn't better for your use case than previously.
It's also not very hard to compile your own kernel if you really need an exact version. I do that sometimes on Fedora when I'm testing a driver I maintain. You can also just keep the RPMs from either previous releases as well (that's not a bad idea so even if you're building your own kernel you can extract the config file from it).
The original value proposition of CentOS was that it was a bug-for-bug compatible, drop-in replacement for Red Hat Enterprise Linux.
> Unless you are doing kernelly things then the small difference in kernel version will not be that big a deal. > It's also not very hard to compile your own kernel if you really need an exact version.
It isn't about having a specific version because the RHEL kernel backports features anyway. If you are compiling the RHEL kernel you might as well use the actual distribution that has been tested with it. What would be the point otherwise?
CentOS had to reverse engineer code drops from Red Hat just like Oracle or Amazon. Oracle will sell you a support contract claiming "binary compatibility" even when they are using a newer kernel than what's in RHEL.
They are trying to make lives easier to collaborate with their frienemies who all maintain a RHEL clone. That means calling CentOS what it actually is: upstream of RHEL.
That's good, because regular old CentOS is not bit-for-bit identical either. CentOS might strip out all the RHEL trademarks, but they had to reverse engineer RHEL code drops just like Oracle, Amazon, Google, Facebook, SuSE, etc.
> The upstreams serve me well enough.
What you don't appreciate is that the distinction between upstream/downstream is a broken metaphor across small time scales. Bug fixes and backports will show up in any number of RHEL clones from Oracle|Amazon|Facebook|Google|Baidu|Samsung before making their way into Fedora|CentOS and finally into RHEL. In the future, RHEL will release their own bugfixes to subscription customers before those fixes find their way into CentOS Stream.
That's the meat grinder of a Linux distribution: uptime "guarantees" are just refunds you get to cash in when a large devops team can't fix a problem fast enough.
> The one thing I can’t get over though, is the decision to roll out this change in the middle of CentOS 8’s lifecycle, trimming out what, 8 years or so of support. That seriously stung. This decision should have been made a year or two ago, before CentOS 8 was released.
If you have a kernel that hasn't been updated in over three years and you need very low downtime, you either have a dedicated operations crew or you don't let it touch the internet.
But if you are so cheap that you don't want to pay for bandwidth/build servers yet aren't willing to violate the "developer" license agreement ... then use a RHEL clone from Amazon|Oracle. They are already undercutting Red Hat support packages, why not make them foot the bandwidth bill too?
Everyone else should be celebrating: Red Hat is basically turning CentOS Stream into a spot where they can collaborate with their frenemies and ship more software faster.
Qubes developers for example, have been spread very thin by Fedora's six month release cycle. Switching away from an RPM distro would be too difficult but CentOS lags too far behind mainline (5 years!) to be useful. If Red Hat can step up their game and push out a major release very 3 years and get the "pipeline" between Fedora and CentOS working ... maybe they can win some market share back from Debian and Ubuntu.
Do you have a source for SUSE reverse engineering RHEL code drops? Last I checked they were unrelated.
At the time, did Karsten et al. disclose this to the CentOS community? This language suggests that Red Hat had long-term plans for CentOS and its core team all along, but to my knowledge, the community was not made aware of these plans so that they could take defensive action if needed.
[^1]:https://www.zdnet.com/article/why-red-hat-dumped-centos-for-...
Even Fedora is pretty damn stable. Stream will be far, far more so. I'm planning to run Stream for all but one of my use cases (my home router, which is a SPOF for my internet connection. Not sure what I'll run there yet).
More info about why I think Stream is mostly a good thing (not universally a good thing, but for most use cases it is IMHO): https://freedomben.medium.com/centos-is-not-dead-please-stop...
They are using this fallout as an opportunity to gauge the market. Risky to be sure, but they didn't do it without considering the risks.
From the article:
> I am confident that CentOS Stream can cover 95% (or so) of current user workloads stuck on the various sides of the availability gap. I believe that Red Hat will make solutions available as well that can cover other sides of the gap without too much user heartburn in the endNobody wants to use CentOS. We have to use it because it's the cost-effective alternative to RHEL. And nobody wants to use RHEL either, but we have to, because vendors won't support anything else.
Part of this is our (the users'/businesses') own fault. We allowed our businesses to get away with depending on an OS with no support until we needed it, which is not really sustainable.
What we maybe should have been doing is creating vendor-neutral open standards for the platform, like a standard kernel (vanilla, anyone?) and a standard userspace distribution (vanilla, anyone??). That way any Linux distro could "meet the standard" ("Install package standard-distro-20200816 to use packages and settings aligned with the standard", and then every vendor could support "the standard". Vendors wouldn't need to lock themselves into one distro, and we wouldn't either. Basically a POSIX for Linux. But there's no money to be had in that.
(It is in fact possible to run a Linux distro without patches. It's buggy as all hell, but it can be done.)
And they should have had this all worked out before making the announcement. As it is, they are just promising a bunch of things that don’t yet exist, after they literally just destroyed all trust in them.
I personally think they will be allowing some production use of free RHEL (assuming we get free RHEL. I expect we will but don't want to count chickens before they hatch). Enterprise production I don't know, but personal production or prod for non profits and/or open source projects, I could totally see that being an option. I'm personally excited for the announcement. I hope they get it out soon.
> They are using this fallout as an opportunity to gauge the market. Risky to be sure, but they didn't do it without considering the risks.
Yes you're right, but it wasn't done this way intentionally. I agree they should have coincided these announcement, but it didn't happen that way. They're doing their best to work with what they have, but they certainly didn't intend to stoke the outrage in hopes to study it :-D
More of my thoughts on the whole thing if anybody is interested: https://freedomben.medium.com/centos-is-not-dead-please-stop...
1000s of people actually want to use free-of-charge rhel!
and they're as scared of oracle as they are of ibm
But functionally, I don't think this helps resolve the issue. The issue is the business model, and RH will never make people happy while it attempts to charge for the OS. The basic thing people are after is RHEL itself, at no cost. That feels unreconcilable.