Hacker News Reader

Comment by heywoodlh | Hacker News Reader

My understanding is that since Red Hat restricted the source availability for RHEL, Rocky is using the following loopholes to gain binary compatibility with RHEL:

1. Using OCI images of RHEL (i.e. on Docker Hub)

2. Using cloud server images of RHEL

They outlined their plans for this in June of 2023[0].

Red Hat’s official stance is that downstream distros like Rocky or AlmaLinux should be basing off of CentOS Stream — that would allow all the EL distributions to contribute and use the same OS base[1]. AlmaLinux pivoted to this method[2] — which seems like a more sustainable approach to me.

My opinion: blows my mind that people who value Enterprise Linux would be willing to continue to use a distro like Rocky that actively works against/around the supported route its upstream provider suggests.

(Personally, I don’t find value in the EL ecosystem, I prefer rolling releases everywhere when possible)

[0] https://rockylinux.org/news/keeping-open-source-open/

[1] https://www.redhat.com/en/blog/furthering-evolution-centos-s...

[2] https://almalinux.org/blog/future-of-almalinux/

View on HN

I fail to see what separates Alma from CentOS Stream at this point

The entire Propose of CentOS was Binary Compatibility with RHEL, largely for Vendor compliance reasons, if a vendor of a commercial software product validated RHEL 7 for their software running CentOS 7 was also supported

Running CentOS Stream, or now Alma Linux would not be as their are no longer binary compatible with RHEL.

the Hostility of IBM and Redhat here has lead me to transition 100% of the CentOS Server I managed to Ubuntu Last year... The software vendors that i use all quickly validated Ubuntu as an alternative to CentOS after the original CentOS Stream announcement which made is a easy choice for me..

I will never use RHEL or RHEL based distribution again at this point so I really do not care about the future of Alma or Rocky either

>Personally, I don’t find value in the EL ecosystem, I prefer rolling releases everywhere when possible

if you are running ERP Systems, LOB Apps, or other critical functions that measure their code life in decades not weeks then EL systems are required. I do not want the system to change as the app i am running has not changed. I want Stability, and security not new features or more performance or even new hardware

AFAIK, Alma has committed to continued full RHEL binary compatibility.

Incorrect, according to their FAQ[0] and the original article I linked[1]:

> What does ABI/binary compatible with RHEL mean?

> In July of 2023, we announced (opens new window) that we were shifting our goal from being a downstream rebuild of RHEL to maintaining ABI compatibility with RHEL. For the AlmaLinux team that means that everything from software applications to kernel modules that work on RHEL will work on AlmaLinux, and if they don't we would consider that a bug.

TL;DR: they are using CentOS Stream as their upstream.

EDIT: Sorry, you're correct -- I was thinking you meant their original goal of "bug-for-bug" compatibility with RHEL.

[0] https://wiki.almalinux.org/FAQ.html#what-does-abi-binary-com...

[1] https://almalinux.org/blog/future-of-almalinux/

> For a typical user, this will mean very little change in your use of AlmaLinux. Red Hat-compatible applications will still be able to run on AlmaLinux OS, and your installs of AlmaLinux will continue to receive timely security updates. The most remarkable potential impact of the change is that we will no longer be held to the line of “bug-for-bug compatibility” with Red Hat, and that means that we can now accept bug fixes outside of Red Hat’s release cycle. While that means some AlmaLinux OS users may encounter bugs that are not in Red Hat, we may also accept patches for bugs that have not yet been accepted upstream, or shipped downstream.

* https://almalinux.org/blog/future-of-almalinux/

Our primary difference: a 10-year lifecycle, with a focus on stability.

I'd say that your use case for CentOS might have been vendor compliance, but our user base doesn't agree that's the only use case.

I'd recommend listening to this podcast with Neal Gompa about Red Hat vs IBM at this point:

https://hackaday.com/2023/12/27/floss-weekly-episode-763-fed...

> I fail to see what separates Alma from CentOS Stream at this point

I believe that AlmaLinux's approach is in-line with Red Hat's vision for downstream OS-es. Rocky's is not.

I think a potential benefit to aligning with Red Hat's approach is that both AlmaLinux and Red Hat will be contributing fixes, improvements, etc. to CentOS Stream -- and any other OS-es that base off of CentOS Stream. And, nothing is preventing AlmaLinux from snapshotting their own stable releases -- which allows the AlmaLinux team to test and release stable releases to their users.

> the Hostility of IBM and Redhat here has lead me to transition 100% of the CentOS Server I managed to Ubuntu

I completely agree with this perspective. This is why Rocky's decisions really baffle me: why use an upstream OS like RHEL if you absolutely do not align with its objectives? I feel like it would have made much more sense for Rocky to make Fedora their upstream instead of implementing workarounds to copy RHEL.

> I will never use RHEL or RHEL based distribution again at this point so I really do not care about the future of Alma or Rocky either

I also do not use any Fedora based distribution (typically just running NixOS or Ubuntu), however, I find that I do care about these shifts because Red Hat has tremendous impact on the trajectory of most Linux distributions. And the downstream OS-es' responses have been interesting to observe to me. :)

> I believe that AlmaLinux's approach is in-line with Red Hat's vision for downstream OS-es. Rocky's is not.

What is in-line with users' vision?

At this point, I'm not sure why end users who care about Red Hat stability don't just use RHEL directly. Red Hat has multiple ways to take advantage of RHEL that involve no cost (that weren't available when the original CentOS project was conceived):

- Free developer licenses[0] (great if you're a home-labber, IIRC you can get up to 16, I think?)

- Red Hat UBI images[1]

I feel like most users of AlmaLinux won't see a huge difference between just using AlmaLinux's release cycle vs RHEL -- especially since AlmaLinux will be ABI compatible with RHEL. I.E. apps that work for RHEL should work for AlmaLinux.

So, back to your original question, I don't know what end-users want more of other than as was said by mrweasel in a different comment: "that's not what some people want, they specifically want RHEL, but for free".

[0] https://developers.redhat.com/blog/2021/02/10/how-to-activat...

[1] https://developers.redhat.com/products/rhel/ubi

"users" is such a large category as to be absolutely meaningless. It's kind of like saying "what do Americans think about big tech?"

>I believe that AlmaLinux's approach is in-line with Red Hat's vision for downstream OS-es.

As far as I can tell this is logically incorrect CentOS/Alma are upstream not downstream and red hat's vision of downstream OSes is they don't exist because that takes a nickel out of IBM's pocket.

AlmaLinux matches releases and versions with RHEL. More clarification in this 2-minute video: https://www.youtube.com/watch?v=DNMnajmyLaA

You're correct, I was referring to AlmaLinux's former status of being downstream of RHEL. Additionally, from Red Hat's perspective, AlmaLinux is still downstream of CentOS Stream (and thus a participant of Red Hat's vision for CentOS Stream[0]).

> red hat's vision of downstream OSes is they don't exist because that takes a nickel out of IBM's pocket

I think CentOS Stream actually contradicts this because (I would imagine) it's a lot of work to maintain and greatly benefits the community -- and Red Hat still makes it freely available for other distributions to base off of.

[0] https://www.redhat.com/en/blog/furthering-evolution-centos-s...

EDIT: clarified RH's vision for CentOS Stream and added my thoughts on CentOS Stream

Technically code-wise we're a downstream of CentOS Stream for the most part, but the end result is more of a hybrid because we're targeting RHEL and can match commits to RHEL commits from stream for a lot of it...so yeah.

Formerly we were just 100% downstream RHEL with none of this nuance, as you mentioned.

> if you are running ERP Systems, LOB Apps, or other critical functions that measure their code life in decades

That's really what confuses me about the people who use CentOS or Rocky, they want or need the stability of RHEL, so that they can run these crucial applications, that mostly aren't cheap, yet they refuse to pay for the development of the operating system they run it them?

I really don't see the point in any of these clones, other than as an educational tool. I see the point in RHEL and while expensive, that's the price of developing this platform. You can buy SuSE, Ubuntu Pro or Oracle Linux, if you feel that RedHat is to expensive (perhaps not Oracle), but that's not what some people want, they specifically want RHEL, but for free.

The theory that Red Hat users are supposed to be paying for development of the system is a recent one. What Red Hat used to say is that the software is free and their users are paying for support.

That made a great deal of sense, as of course Red Hat themselves were shipping a great deal of software written by other people without paying for its development.

There's nothing confusing about people declining to go along with this change.

> There's nothing confusing about people declining to go along with this change.

No, but why stay on a platform from a vendor that has a business model you find hostile? I can see the need for a transition period, but I'd start looking for another vendor that has value and pricing that aligns more with my ideals. People seem hellbent on staying on a RHEL-like platform.

Also, CentOS isn't/wasn't exactly new, the current pricing model from Redhat isn't nearly as old as CentOS.

> People seem hellbent on staying on a RHEL-like platform.

Not really, Ubuntu Server has surged in popularity.

> That's really what confuses me about the people who use CentOS or Rocky, they want or need the stability of RHEL, so that they can run these crucial applications, that mostly aren't cheap, yet they refuse to pay for the development of the operating system they run it them?

Yes. Non-prod (or even prod and non-critical) servers usually outnumber critical prod servers, and it makes sense to not run RHEL on them and pay for support, but to run a 100% compatible distro.

> the people who use CentOS or Rocky, they want or need the stability of RHEL, so that they can run these crucial applications

No, the people who need the stability of RHEL were already paying for it. The people who used CentOS were doing so to be able to develop and test software that would eventually be used by people running RHEL.

>> but that's not what some people want, they specifically want RHEL, but for free.

Well in my world we used to run many Dev/Training/Non-Prod servers with CentOS and then Run the Prod Servers with Licensed RHEL

Similar to having a MSDN License for Windows vs Production licenses

Then with the announcement of CentOS Stream they tried to sell their Dev program, but that program was TERRIBLE for enterprise use, it was built for individual developers, and not at all compatible with the model many organizations where using. They have since tweaked the terms to make is somewhat more compatible but it still IMO, not very useful for many of the scenarios I found myself in over my life

And frankly I have enough problem keeping track with MS Licensing I do not need the additional headache I might as well just use windows at that point.

Red Hat was able to become a billion dollar company by providing great support, and being easy to do business with, even before IBM buyout and accelerating after their support was becoming less and less quality, and they became more and more hostile / harder to do business with

Many organizations where already questioning their continued use of Licensed RHEL because of this, this like played a part in them killing CENTOS (and yes they killed CentOS, CentOS Stream is not a compatible replacement) as that was easier that fixing the systemic internal issues around support and business processed.

To be fair, rocky isn't on its own here. https://openela.org (CIQ=Rocky)

> I fail to see what separates Alma from CentOS Stream at this point

Alma is getting their source packages from CentoOS Stream, but they are using the specific versions that are in the Redhat releases they are targetting. They aren't just rebuilding Stream sources from HEAD.

IBM's official route results in a distro that is not bug-for-bug compatible with RHEL, at least not without a lot of unnecessary work. The route Rocky is taking results in a distro that is bug-for-bug compatible.

> The route Rocky is taking results in a distro that is bug-for-bug compatible.

Point taken.

From my perspective, it seems like a short-sighted approach to rely on loopholes to obtain bug-for-bug compatibility. I know I wouldn't be comfortable running anything production-grade on Rocky Linux with their current approach.

I believe if Red Hat had legal grounds to sue Rocky, they would have done so already. And there's no possible way for RH to obfuscate their sources so Rocky will continue releasing an exact RHEL clone unless Red Hat sunsets RHEL.

So what? Doesn't matter how convoluted the "loophole" is. There will always be a loophole because Red Hat must leave one open, because this is open-source software and it's impossible for them to truly close all avenues to using your RIGHTS under the license.

We should he celebrating people working around DRM, not being fearful of it.

Fair enough! As I stated, I have no personal stake in Rocky's approach as I am not a user interested in the Enterprise Linux ecosystem. That being said, if I was, I would be concerned with the approach as I can't see it being sustainable long-term to rely on the workarounds.

(I'd be happy if the Rocky Linux folks prove me wrong -- the more variety there is with Linux, the better!)

>> Red Hat’s official stance is that downstream distros like Rocky or AlmaLinux should be basing off of CentOS Stream — that would allow all the EL distributions to contribute and use the same OS base

That stance goes contrary to the "everyone benefits" nature of free and open source software. It turns "everyone benefits" into "Red Hat / IBM profits". Why should the community contribute to Red Hat's bottom line if Red Hat refuses to contribute back to the community?

Rocky Linux's "going around Red Hat's back" is only necessary because of Red Hat's choice to not contribute back to the community.

If IBM wants to expand its AIX (https://en.wikipedia.org/wiki/IBM_AIX) business, it can do so without tainting Red Hat.

Accusing Red Hat of not contributing to the wider free software community is so off-base I’m not sure how to respond.

Is there a single large FOSS project who’s git history isn’t filled with @redhat.com ?

Oh, so is it okay to share RHEL sources back with the original project communities or not?

If you share RHEL sources with the original project communities, what happens?

>> Is there a single large FOSS project who’s git history isn’t filled with @redhat.com ?

Does this give Red Hat the right to effectively "close source" the code for RHEL despite contributions from the rest of the community?

> Does this give Red Hat the right to effectively "close source" the code for RHEL despite contributions from the rest of the community?

Legally, yes. Not sure what the point is you're trying to make.

I'm not sure why anyone would want to use a downstream distribution of Red Hat if they don't like Red Hat's trajectory. If you don't like Red Hat's current objectives with RHEL, it should be as simple as: stop using something downstream of RHEL.

> Legally, yes. Not sure what the point is you're trying to make.

Legally, no; unless RH is the only copyright holder, they get to follow the same rules as everybody else, and for GPL packages that means not imposing additional restrictions on redistribution of source code.

> If you don't like Red Hat's current objectives with RHEL, it should be as simple as: stop using something downstream of RHEL.

People are fine with being a downstream of RHEL. It's being upstream of RHEL that's the problem; Stream is RHEL Beta by another name.

>Legally, no; unless RH is the only copyright holder, they get to follow the same rules as everybody else, and for GPL packages that means not imposing additional restrictions on redistribution of source code.

And there aren't any. "RHEL Linux" isn't GPL licensed, the kernel is, and glibc is, and systemd is, etc. Individually.

The sources for all of those packages (and ones that aren't GPL licensed, for which there is not actually a legal requirement to distribute sources), individually, are available via CentOS Stream just as they ever were.

What Red Hat has stopped doing is providing a git repo containing the exact combination of package sources which collectively make up one particular version of RHEL. Instead, all of the sources are still available, but if you want to rebuild the entire distro you need to figure out which particular versions to use. And that's exactly what Alma Linux does.

> "RHEL Linux" isn't GPL licensed, the kernel is, and glibc is, and systemd is, etc. Individually.

That is why I said "GPL packages", yes.

Okay, so if I buy RHEL, and I download the SRPM for the kernel, and I publish it - which I'm legally entitled to do, what with the kernel package being GPLv2 - is RH not going to terminate my RHEL license?

Almost certainly not. I don't think that has even happened with Rocky, for whom they're not just downloading one package.

Yes, you can. This is something everyone gets mixed up.

RHEL is built from CentOS Stream sources. The changes Red Hat made to RHEL source distribution mostly just make it more challenging to reconstruct an entire build of RHEL from CentOS Stream - because you would have to map backwards from all of the different versions of sources available in CentOS Stream to the exact versions that a particular version of RHEL happens to use. But at the scale of any individual project, it's trivial for any upstream community to just look at the latest CentOS Stream sources for one particular package, if they want to look at which patches Red Hat is applying. There are no legal or contractual restrictions on this whatsoever.

This is setting aside the fact that most patches are backports from one upstream version to another, and bugfixes that get submitted "upstream first" to begin with, which makes the whole discussion kind of moot. The typical scenario for a net-new bugfix would be that a Red Hat employee submits the fix upstream and gets it reviewed and merged by said upstream months before it ever sees the light of day in RHEL, so upstream maintainers would never need to look at the RHEL sources in the first place.

Your analysis seems to be ignoring all of the work required to create and maintain the CentOS Stream / RHEL distribution in the first place, which is substantial.

Prior to CentOS Stream, if you were a CentOS user or maintainer who experienced a bug, your only option was to file a bug on the RHEL bug tracker and wait for a Red Hat employee to fix it for you. It was the Android-esque "throw the code over the wall" model of open source.

Post CentOS Stream, a CentOS user or maintainer files the bug in the CentOS Stream bug tracker, can write a patch and have it reviewed by other users/maintainers/Red Hat employees tracking the issue, and then benefit from Red Hat QA'ing the fix.

So counter to your claim, I would argue that the CentOS Stream model provides more bidirectional / mutual benefits.

>> Your analysis seems to be ignoring all of the work required to create and maintain the CentOS Stream / RHEL distribution in the first place, which is substantial.

Substantial enough to effectively "close source" RHEL?

Red Hat's changes (https://www.redhat.com/en/blog/red-hats-commitment-open-sour...) effectively converted RHEL from a fully open source Linux distribution to "source available to paying customers only" proprietary product.

The changes are effectively a violation of the GPL by prohibiting customers from distributing RHEL sources through Red Hat's Terms of Service and EULAs:

https://rockylinux.org/news/keeping-open-source-open/

RHEL is not "effectively closed source" by any stretch of the imagination. The sources are available as CentOS Stream -- from which RHEL is built in the first place!! It is true that, in order to put together an exact copy of RHEL, you would need to do a bit of work to figure out exactly what versions RHEL uses, and specifically pick those versions out of CentOS Stream. It isn't as simple as downloading the entire tree of sources from a single git repository anymore. But that's not the same as "effectively closed source", either.

If you provide GPL software to your customers, you have to provide everything they need to build this exact version yourself, not point them at a repo and tell them to figure it out.

Red Hat does provide all of that to RHEL customers. This is a discussion about what non-RHEL customers get.

Even non-customers are provided with all of the sources they they could use to build RHEL, given some additional effort. It's less trivial than it used to be, but not particularly difficult.

Alma takes the approach of reconstructing RHEL using these public sources, Rocky takes the approach of buying AWS VMs running RHEL for a few minutes at a time so that they can download RHEL sources as a "customer" while only paying a few dollars a month to do so.

The point is, even non-customers can rebuild RHEL using sources made publicly available by Red Hat if they so desire. It's not "effectively closed source", even to them.

Prior to CentOS Stream, couldn't you file the bug in the upstream project's bug tracker instead, and then still do all of the other stuff you said?

You'd still have to wait for someone at Red Hat to backport the fix themselves. There wasn't really a way to actually participate in the engineering of CentOS itself.

Won't you still have to wait for someone at Red Hat to move the change from Stream into RHEL?

You'd have to wait on it to be QA'd before it would get into Stream officially, but once it's in Stream the process of it getting into RHEL is mostly automatic. It would just be a matter of waiting, since RHEL only does releases every 6 months whereas Stream releases constantly.

With CentOS you would have the same problem, except that there's no way to make the engineering part happen any faster than Red Hat's staffing and priorities allowed.

Reply on news.ycombinator.com