Hacker News Reader

Comment by tadbit | Hacker News Reader

Given what Red Hat and the CentOS project did to CentOS 8 I have no desire to use RHEL ever again. Right now it's just Ubuntu LTS and Debian for my needs. Working on eliminating RHEL at work as well, wherever I can.

View on HN

Yep. We got burned out on the lies RHEL8 pushed to serious orgs (utilities, gov agencies, etc.) around podman readiness & replaceability over docker. I'm glad people are getting their bonuses and good for docker to have competition, but don't abuse your trusted advisor & monopoly position to mess with societal infrastructure and the admins keeping it going. Before we promoted other OS's for our AI users as a matter of general GPU readiness but were game for supporting RHEL, but now we actively recommend against it as too untrustworthy going forward.

I feel you could run docker on rhel 8. How was having an option worse?

What is a few clickthroughs for the owner of a home box or regular co box can be weeks for teams or even a deal breaker in critical envs that are locked down. The money people know where the money comes from when making strategic decisions of what to make easy vs hard.

They made docker hard for regulated environments while making their broken competitor built-in and then marketed theirs as a replacement. This incurs all sorts of costs in schedule + $$$ + reliability where teams are pushed to figuring out if podman works in their case ("why wouldn't it?") + when not, start over with an unnecessarily complicated round of change management steps for enabling docker from centos7.

RHEL/IBM are allowed to use their trusted OS position to be anti-competitive and overall non-neutral for above-OS layers, and to the clear harm of customer. But I am also allowed to say we shouldn't trust & tolerate such a provider for vendor-neutral infra in regulated environments. Secure infra is important and podman is pushing docker on important areas here, so it's been disappointing to see the one-step-forward two-steps-back.

I spent a fair bit of time trying to move to the podman ecosystem and rootless containers before deciding it wasn't ready for production in RHEL 8. I was used to RHEL being more, well, stable and wasn't expecting them to be pushing premature software (that is Fedora's job), so I was disappointed at the state. I would probably have been more upset if RedHat's misleading marketing had convinced higher-ups like the CTO, and I had to push back against directions from above, but investigating it and rejecting it was solely my decision so it wasn't a big deal.

At least Debian cannot and will never be bought, will always be there, has an army of contributors (some of them kernel developers), and is basically the only enterprise grade free distribution.

Now just add some RHEL/CentOS containers on top where needed and forget about the RH lock-in.

> RHEL/CentOS containers ... RH lock-in

So if a non-IP-lawyer reads the redistribution terms of the RH Universal Base Images, there are some very dubious implications in there, such as #22 and #42.

  https://developers.redhat.com/articles/ubi-faq#

Has anyone done a third party analysis of their EULA? My org, an ISV, may need some legal cycles to avoid stepping in this trap.

I'd be just fine avoiding UBI because of that, but there are some orgs whose security posture demands only UBI images are allowed in their domain so ISV's may be forced to pay to play.

#22 and #42 is how they try to prevent people from adding RPMs from mainstream RHEL. I have seen GitHub repos owned by RH employees that have containers files that require you to attach entitlement cert and the Red Hat cdn repo to install additional packages. It doesn’t violate EULA because they aren’t distributing the image.

RH knows people will try to abuse the free UBI image and install RPMs they shouldn’t be.

I work with Red Hat Partners so I know these rules well.

Yeah, they are base images devoid of copyright/trademark assets like artwork or whatever, and so it goes... just a platform to layer around. Like you wrote the first thing many people want or need, is to layer other RHEL packages, or if the software lacks many dependencies then the ISV just integrates their own software over top, then packages that layer themselves...

Meh

> I'd be just fine avoiding UBI because of that, but there are some orgs whose security posture demands only UBI images are allowed in their domain so ISV's may be forced to pay to play.

That seems fine IMO; use free OSs wherever possible, but if customers want RHEL just pass the cost through and let them deal with it.

It would be quite nice to see Debian adopt a longer term release model. In a lot of environments that was the deciding factor for RHEL/CentOS. A stable OS with security support for a decade.

The current Debian release cycle is closer to 2 years which means a big increase in testing and OS upgrade cycles.

https://wiki.debian.org/LTS ?

Yes, LTS support is 2 years.

Debian releases are supported for about half the length of RHEL releases overall, LTS included.

Debian “Stretch” for instance was released in 2017 and support ends 2022, so a 5 year cycle.

RHEL on the other hand has a 10 year cycle, and offers extended support beyond that:

“Red Hat Enterprise Linux Version 8 delivers a ten year life cycle in Full Support and Maintenance Support Phases followed by an Extended Life Phase.”

https://access.redhat.com/support/policy/updates/errata/#mas...

With Debian upgrades are realistically being planned every 2-3 years as the release approaches LTS.

That makes for a lot of extra upgrade work (about 3x more) when compared to the 10 year RHEL life cycle.

Oh, I see. Yes, that's true, although it would take a rather lot of resources; for a decade of support, you basically need devs who can support each package in the distro independently of upstream. Still, if you're willing to pay for it I expect there are companies who will offer that support.

That, or, package maintainers could decide to maintain their packages for 10+ years. That at least distributes the workload among the package maintainers. Unfortunately, most package maintainers don't like spending time backporting bug fixes etc. and would rather be making shiny new releases (and can you blame them?)

> and can you blame them?

Exactly; almost nobody wants to maintain old versions like that. For most people, the fun part is writing new stuff, and maintaining old stuff without breaking backwards compatibility is particularly annoying to put up with. I certainly couldn't blame anyone, particularly who was just publishing software for free, for not wanting to deal with all that.

Amen! I share these thoughts and plan of action completely.

I'm completely out of the loop (I only use Debian). What did RH do w/ version 8?

Eliminated it in favor of a less stable rolling release edition. https://www.zdnet.com/article/centos-linux-8-is-about-to-die...

CentOS 8 was supposed to be supported into 2029. Approximately a year or so after being released that was changed and would only be supported into 2021.

Many people in the CentOS community, even RHEL employees, tried to play it off like they never meant to support it so long, but some time after the announcement they confirmed they did indeed strip away 8 years of the planned support.

This burned a lot of people that had already started deploying systems using CentOS 8 into Production. A lot of time and money has been spent by many people, organizations, and companies rectifying this situation.

Additionally, CentOS has become "CentOS Stream" and no longer is 1:1 with RHEL and considered stable. It's an in-between Fedora (bleeding edge) and RHEL (boring and stable), which means things might change or break, etc... which is sort of the opposite of what your typical CentOS user wanted. Really a missed mark from IBM/RH.

This led to several CentOS replacement distros, including Rocky Linux (made by many original CentOS people), AlmaLinux and others. Both Rocky and Alma are supporting 8.x for 10 years, prompting many CentOS users to switch over.

As I understand it, this is an incorrect representation of what Stream is. Unfortunately I rarely see this misconception corrected.

Stream is only "between" fedora and RedHat until RHEL X.0 is released.

For instance, once RHEL 9.0 development branched from fedora 34, Stream 9 became the upstream and just ahead of all further RHEL 9.X releases, and doesn't depend on fedora anymore.

Furthermore all Stream rpms undergo the exact same RHEL testing and quality validation chain than proper RHEL packages.

As such by using Stream 9, you are arguably receiving the bug fixes that are eventually going into RHEL 9 proper, just in advance a bit.

RedHat engineers have argued that keeping up to date would amount to effectively running a faster fixed (less buggy overall) OS than RHEL.

> RedHat engineers have argued that keeping up to date would amount to effectively running a faster fixed (less buggy overall) OS than RHEL.

Wow, what a remarkable offering; they give free users even better service than their paying customers! It's so benevolent of them to give such amazing service away and definitely not use this totally-better product as a beta to ship updates before it gets to the stable users.

Edit: Seriously, though; if a company has 2 almost-identical products and claims that the free product is equal or better than the free product, they're either lying to the free users or defrauding the paying users.

The people who pay for RHEL want the stability. So a faster-updating OS is technically superior but requires more regular testing & development work from the customer side.

The people using CentOS also wanted the stability. And if RH had added Stream as an option, that would have been totally reasonable and a great move. It was the way they killed the stable version while proclaiming that Stream was totally a good replacement that was disingenuous, because it was not a replacement for "as stable as RHEL without cost".

I've never seen anyone use this logic to recommend installing Windows Insider on production servers.

Yeah, Red Hat is actually moving the black box of releasing-engineering out in the wide open, for anybody to replicate, and see how the sausage is made.

CentOS Stream would be a perfectly suitable alternative to RHEL9, and in many ways more desirable. And, in a strange way all those people who were being super conservative... more attracted to centos being behind RHEL, as if that made CentOS more stable by being slightly behind... well, that's RHEL now. So.. uh.. maybe just use RHEL instead, since it's now even more conservative? Also, it's important to recognize during all the drama, Red Hat changed their licensing to be pretty much free for small to medium sized operations. So the argument of being cheap, or whatever free value of CentOS is now entirely gone... Red Hat only cares to license from the medium to large.

One thing that hasn't been clear to me about CentOS Stream is if it is possible to determine that the repos at this instant are identical to a given RHEL minor release (and possibly stay there temporarily). For example, if I have a non-production system running CentOS Stream 8 so I can test out changes to RHEL earlier, can I be confident that testing on a certain date will be applicable to the next RHEL release when it comes out, or do I still need to test on RHEL separately when it is released?

I believe the answer is yes, but I'm not 100%.

My understanding is the the stream repo sorta rolls forward but leaves behind a trail of X.Y.Z checkpoint repos representing major.minor.batch-number, so that would be like rhel-9.1.0 for the zero-day batch of updates on rhel-9.1

This is not true at all.

CentOS Streams is the next X-Release of RHEL. I work with RH Partners and we use Streams as a test platform before we move to QE on Beta and RC builds.

This has been refuted by RH many times. You want to be upset they took away CentOS, that is your choice but spreading incorrect information about Streams is simply misleading.

> It's an in-between Fedora (bleeding edge) and RHEL (boring and stable), which means things might change or break, etc...

This is a misleading explanation.

CentOS Stream 9 stays between RHEL 9.X and RHEL 9.X+1.

Once branched from Fedora CentOS Stream follows RHEL ABI compatibility of that specific major version and doesn't consume Fedora updates.

It is also tested together with RHEL.

For the few systems I upgraded to CentOS 8 back before it was killed, I switched them to Rocky Linux (Alma's also a good choice).

I'm still waiting a bit longer to see whether I'll keep my toes in the RHEL-ecosystem-waters, or if I finish moving everything to Ubuntu LTS and Debian.

I think the work being done with Rocky and Alma linux is great! Many kudos to them.

However, I'm worried history will simply repeat itself or RHEL will make downstreaming difficult in some way, which could kill those projects.

My understanding is that CentOS as we once knew it (the freely-distributable rebuild of EL 7 and earlier) withered away because its creator and primary maintainer was hired by Red Hat. There was a clear conflict of interest at that point, and Red Hat probably wasn't going to keep paying him to maintain a project in perpetuity that ate into their bottom line.

https://www.linuxfoundation.org/blog/centos-project-leader-k...

So it's not so much that RHEL tried to strong-arm third-party rebuilders of Enterprise Linux - and I don't think it's in their culture to try - but people involved in these projects are always at risk of being bought out. Volunteering for open source can be a lot of work.

I just switched to Alma Linux, it was nbd.

How is it so different? It's still less bleeding edge than Fedora.

It always makes me feel people are making such a big fuss about it being slightly ahead of RHEL than slightly behind which has the benefit of having fixes earlier than RHEL.

If you got a problem, you could always pay for RHEL.

Giving up completely on technical competence and using a properly maintained distro? Debian/Ubuntu may be useful for harmless hobby users, but certainly not for anything serious.

IMHO Debian and Ubuntu are more harmful than Redhat

Reply on news.ycombinator.com