Hacker News Reader

Hacker News

Comment by parasense | Hacker News Reader

Full thread

parasense

To quote Jeff:

> Here's how it used to work:

> Red Hat would grab a copy of Linux They would add magic sauce that makes it Red Hat Enterprise Linux

> They would release a new version.

> They would update a source code repository with all the data required to build it from scratch

A few remarks. The "add magic sauce" part is gone, all the secret sauce, or black box machinery is out in the open in CentOS stream. Red Hat no longer does any RHEL development behind the corporate firewall, and does everything out in the open in CentOS Stream. There is one caveat for security CVEs, but effectively those go into CentOS Stream and RHEL at the same time when the CVE embargo expires.

What this comes down to is wilfully ignoring CentOS stream. All the clones of RHEL can simply track CentOS stream, and even replicate all the "Secret Sauce" that is the CentOS Stream CI/CD workflow. The process of replicating RHEL is so much easier today than ever before.

View on HN

Two thoughts:

> The process of replicating RHEL is so much easier today than ever before.

Doing so bug-for-bug requires more effort than it did a few weeks ago. And they're basically doing that work in parallel with Red Hat itself (duplication of effort).

Also, making a change like this (forcing Rocky and AlmaLinux to change in the middle of the 9.x release cycle), when the previous change (killing CentOS) was _also_ in the middle of the 8.x release cycle, with not even 24 hours warning... makes me feel this change was not _only_ made "to make CentOS and RHEL development better."

If that were the case, the changes would be announced with at _least_ weeks (if not months) of warning, so the community could plan for it instead of getting blind-sided... now twice in a row.

Why do bug-for-bug with RHEL at all? Why not turn Alma and Rocky into stable distributions in their own right?

If this pushes RHEL into oblivion, why should anyone care?

The Debian ecosystem isn't centered on Ubuntu, it's centered on Debian. Comparisons between the two ecosystems are invalid because of this simple fact.

It would be nice to see the "Red Hat" ecosystem recenter around something which isn't a direct source release of a commercial product. Alma and Rocky should rebase on CentOS Stream for starters, and then try to get closer to Fedora as an upstream.

(BTW, I speak as someone who did corporate systems administration for years and understands the use case of replacing RHEL licenses with free CentOS installs perfectly.)

> Why do bug-for-bug with RHEL at all? Why not turn Alma and Rocky into stable distributions in their own right?

Because the whole point is for them to be 'compatible'.

Many of the users are running software that needs to be on RHEL/CentOS/Rocky/etc or it won't be supported.

Sure, there are users who just want a stable OS, but the real point of Rocky/Alma is to have a 100% RHEL-compatible, but free distro.

There are many levels of compatibility. ABI compatibility is perhaps the least stringent, bug compatibility is the most picky.

I'm not certain that much more effort is required.

Untraceable uploads with Onionshare over a Tor browser would conceal the origin of source RPMs that did not bear fingerprints.

Once captured, all that is really required is the updated SPEC file, and any modified sources or patches (if I remember my RPM internals correctly).

Alma could maintain such an Onionshare instance. If the SPEC and patch files are specifically GPL, I'm not sure that Alma could be compelled to forego them.

Let me preface this by saying, Jeff owes Red Hat nothing and he's free to stop "supporting" RHEL for any or no reason at all.

That said, "grab a copy of Linux" seriously, gloriously, hilariously handwaves away so many things that Red Hat does to create a RHEL release.

Red Hat pulls together hundreds or thousands of upstreams to create RHEL, participates in many of them, tests all that together, helps partners certify software against it, etc. etc. etc.

It's true, Red Hat doesn't "own" the Linux kernel, but it's done a ton to help develop it over the years. But RHEL is not merely the kernel nor any single upstream. RHEL is a product that comprises thousands of packages all tested together and then released as a supported product.

What Red Hat is trying to guard is not the source code to any single or even groups of projects. It's trying to preserve and capture the value it created from all those parts. Coincidentally, that value is what businesses, competitors, and the community are clamoring for and not the source code alone.

They want that single point-in-time snapshot that everybody agrees on as a de facto standard because the overall community has never been able to agree on another workable standard that would allow targeting applications across the board. And that standard has a name, and it's RHEL. And it belongs to Red Hat.

You can have all the pieces and assemble them yourselves if you like. But nobody is entitled to certify it as (officially or unofficially) RHEL except Red Hat.

If that angers you, I heartily encourage folks to build Debian up as the standard we all certify against. Or start your own business that overtakes Red Hat and earns the place RHEL has today.

Mark Shuttleworth took potshots at Red Hat's business model with RHEL for years, and it seems to me that they're doing a very similar thing now with Ubuntu Pro by holding back updates to packages after 5 years and charging for updates to the Universe and Main repos.

I think the main thrust of all of this is to de-commodify Linux. To make RHEL really be in people's minds it's own operating system outside of Linux. It's hard to do when it's really just a ball of generic open source components. Hence, this hostile behaviour. They don't want to be Linux, they want to be RHEL and ask not about the man behind the curtain.

Just the opposite, I think. Red Hat is fighting the idea that the operating system is a commodity, and demonstrating that the RHEL subscription does have value.

Red Hat's competitors have tried really really hard to undermine RHEL as something of value. The cloud providers like AWS and Azure provide their own base Linux distros to run workloads on. They'd love to cut Red Hat out of the picture. Oracle wants to convince customers to give it all their money to run workloads, hence Oracle Linux -- that's based on RHEL.

If RHEL were, as you say, "a ball of generic open source components" then you wouldn't see all this wailing and gnashing of teeth about making it harder for Alma and Rocky to claim "bug for bug" compatibility with RHEL. Nobody would care.

Note that this has been going on for more than 20 years, and this isn't the first time. I've been writing about this on my blog, but we've seen this story before and the only thing that has changed are the names of the vendors / projects and the version numbers. (See: https://dissociatedpress.net/2023/06/26/red-hat-and-the-clon...)

Not the opposite. You're agreeing with the parent that Red Hat is viewing to de-commodify Linux. You just can't seem to see why this a bad thing.

Red Hat does not have the interests of the community in mind any longer, which was inevitable when IBM purchased them.

The parent comment is a bit confusing in that it says de-commoditizing Linux, which is one thing, but then talks about de-commoditizing RHEL.

Yes, Red Hat wants to show that Linux can be treated as a commodity except when it can't; that the work on a distro can be separated into parts that can be shared with the community and parts that people are willing to pay for; that rebuilders are reducing the value of the latter and the only game-theoretical outcome is that no one will have the money to do this work.

Without going into the discussion of whether that's correct or not, it's not a particularly new stance. It was the whole point of separating Fedora and RHEL in 2004.

I agree with most of what you said, but:

> If that angers you, I heartily encourage folks to build Debian up as the standard we all certify against. Or start your own business that overtakes Red Hat and earns the place RHEL has today.

This strikes me as a variation of the "if you don't like Apple or Google's rules, make your own phone" or "if you don't like Youtube's content policies, then start your own video company." It's not necessarily wrong, but it's wildly impractical. It certainly may come down to that, but it would be far better for everyone involved to reach some sort of happy medium point that doesn't result in mass duplication of effort and fragmentation. Before telling people to just fork the project if you don't like it, it would be great to address people's concerns as much as possible, and that can't happen unless people make their concerns known.

It's also, as has been demonstrated, wildly impractical for Red Hat to continue exactly as before without others undermining its business.

The thing is that everybody is only lobbying Red Hat to make concessions, but nobody's going after Oracle or Rocky or Alma or other clones to say "hey, cool it. Red Hat's trying to keep a business going here. You're undermining their business. If you keep this up, they're going to make it harder for everybody to use a free RHEL for non-business situations."

The difference, too, is that I (probably) have a business relationship with Apple or Google if I'm complaining about their wares. When I complain about iOS or Android it's because I've purchased one of those phones (or want to) but it has a defect (in my opinion) or user-hostile anti-feature preventing me from getting the value I paid for.

This isn't that. This largely seems to be people who are not paying for RHEL demanding Red Hat allow them to continue not paying for it but receive its value.

They could also shrink down to a more manageable business unit... except IBM paid $34 billion for Red Hat, and you can't ever decrease revenues or the axe drops :(

Well, remember too that IBM didn't pay $34b for just RHEL - a big part of the purchase was OpenShift which has been coming up in revenue and has RHEL as its underpinning. And I have a lot more to write and say about that in a series for my own blog, but certainly IBM isn't looking for RHEL revenue to go down...

And to be very fair to all involved, that's not even remotely unique to IBM. IBM is a public company and the expectation for public companies is that revenue always goes up. Always.

Red Hat had to live with that as an independent public company too. If IBM weren't in the picture and Red Hat were reporting poor subscriber numbers for RHEL, investors would be calling for heads because late stage capitalism. There's no more room for long-term thinking if that doesn't involve "numbers go up, every quarter."

Yeah, and regarding OpenShift, I wonder how well that play has been working for IBM. My guess is they were expecting to 10x or 100x OpenShift revenues as giant corporations standardized on containerized infrastructure...

...but then Kubernetes (sans OpenShift) seemed to do the 10x'ing while OpenShift was just what RH customers used if they weren't already exploring Kubernetes through some other cloud provider's offering.

"I wonder how well that play has been working for IBM."

In January IBM reported that OpenShift is pulling in $1 billion in Annual Recurring Revenue [1]. OpenShift existed as a product before Kubernetes but was rebased on Kubernetes with v3. That was released (I think) in 2016? So that's basically 6 years from $0 Kubernetes revenue to $1 billion.

Red Hat numbers overall are sort of blended in with the rest of earnings for their group, I think, so I don't see exactly what Red Hat's numbers are now overall. When I worked for Red Hat they'd break it out internally but I wouldn't be able to share that unless it was reported publicly...

It took Red Hat something like 12 years from IPO to $1 billion in ARR, as an entire company. Not sure when RHEL alone became a $1 billion ARR business, but you have to remember that Red Hat's numbers included JBoss and other software too when they hit $1 billion.

As far as I know, none of the cloud providers break out Kubernetes numbers alone. I don't know, for example, how much AWS racks up via its Kubernetes. Of course, AWS also gets some of the OpenShift business since some of Red Hat's customers are on AWS, there's ROSA, etc. (Then again, AWS also gets a slice of RHEL too...)

Worth noting that Google Cloud posted its first profitable quarter in April of this year. No idea what Google's market share for Kubernetes is, but if I had to guess I'd say that their distribution isn't #1. (If anyone actually can accurately measure this, anyway...)

If IBM expected OpenShift to have grown beyond $1bn by January this year, I'd say that was unrealistic. I mean, obviously because it didn't, but also because when IBM acquired Red Hat officially the total Red Hat revenue was about $3.4 billion and if I'm not mistaken "emerging technologies" (including OpenShift) accounted for only $225 million. [2] If OpenShift was, say, $100m of that then they have (in fact) 10x'ed it.

And, finally, I don't think OpenShift revenues reflect whatever services IBM has packaged for OpenShift as a standard deployment platform. So - if IBM has made it easier to sell other things because they only have to target OpenShift, and they're successfully pushing OpenShift into major customers, then it's a pretty good deal.

But for now they also need to protect their primary Red Hat revenue stream, and that still appears to be RHEL.

[1] https://www.ibm.com/investor/att/pdf/IBM-4Q22-Earnings-Prepa...

[2] https://www.theregister.com/2019/03/26/red_hat_2019_results/

>There's no more room for long-term thinking if that doesn't involve "numbers go up, every quarter."

That is obviously unsustainable. I get that they need to make a profit but there's no fundemental reason numbers have to keep going up and at some point they won't be able to go up anymore, even if the business remains profitable.

Yeah it is obviously unsustainable, but nobody knows how long it can last, and they don't really care. Whoever gets to helm the ship when the end starts will get fired by the board, and somebody new will come on board. When they eventually can't grow, they'll start acquiring other companies and using that as their growth. Repeat forever.

> It's also, as has been demonstrated, wildly impractical for Red Hat to continue exactly as before without others undermining its business.

> The thing is that everybody is only lobbying Red Hat to make concessions, but nobody's going after Oracle or Rocky or Alma or other clones to say "hey, cool it. Red Hat's trying to keep a business going here. You're undermining their business. If you keep this up, they're going to make it harder for everybody to use a free RHEL for non-business situations."

You make some really great points here. I think you've mostly won me over.

> It's also, as has been demonstrated, wildly impractical for Red Hat to continue exactly as before without others undermining its business.

How was that demonstrated? RH became, famously, a billion-dollar company under the old model; why would continuing that not work?

CentOS never offered paid support. Rocky and Alma are trying to clone the whole package.

With that being said, I do think it falls into the category of "problems that probably wouldn't exist if CentOS was still around"

Rocky and Alma are new, but Oracle's been doing OEL for a long time. Although I'll grant that adding more players to the mix may have exacerbated things, or at least giving RH the impression that things were getting worse, which I agree is rather a self-inflicted problem since I suspect the logic was "RH killed CentOS, so we need to replace CentOS, which takes resources, so we'll sell optional support for our clones".

Yup, there is a market for cheaper support. The problem is RHEL licensing (price and hassle). If CentOS was around, RH could offer cheaper support for it. That was the solution to this problem, not kneecapping their platform.

> This isn't that. This largely seems to be people who are not paying for RHEL demanding Red Hat allow them to continue not paying for it but receive its value.

Well... I am not so sure about that. As I wrote above, it's actually easier to make a copy-cat of RHEL today by leveraging CentOS Stream than anytime before. So it's not about costs, but perhaps more about the inconvenience of switching off the old workflow for the new hotness.

Red Hat has no technical self interest in perpetuating an old legacy service considering that CentOS has itself migrated to CentOS Stream. Please remember that maintaining that CentOS git repo was really a self-serving activity, to help CentOS project rebuild RHEL packages using the clunky old workflow. In reality supporting the old workflow was never an act of charity, Red Hat did that for whatever business reasons following the CentOS take-over. Certainly the old workflow benefited other folks beyond CentOS community, namely the other copy-cat rebuild distributions.

And certainly, as you have argued, there is the side-effect of the old legacy service benefiting copy-cat distros, and by extension the cost-free user community. But as I argue, not necessarily opposing your view, but offering a counter point... it's more of an inconvenience that some folks seem to wilfully or genuinely misunderstand. The misunderstanding is that Red Hat is killing copy-cat distros, and they are not. Super smart & motivated people will still copy-cat Enterprise Linux, just like they always have. And certainly, Red Hat probably has near zero interest in helping any competitors rebuild packages, or compose the distro. But it's a stretch for folks to argue that ending one workflow in favor of another is slapping the faces of loyal free-loaders. The nexus of ideas/arguments from the one starting point to the wrong conclusion is just tenuous at best, so I'm not surprised some people would construct a straw man argument narrowly focused on short-term outcomes. Folks just need to get on-board with CentOS Stream, or setup some other way to rebuild EL packages. It's not that hard to reason about this topic.

> Red Hat pulls together hundreds or thousands of upstreams to create RHEL, participates in many of them, tests all that together, helps partners certify software against it, etc. etc. etc. > It's true, Red Hat doesn't "own" the Linux kernel, but it's done a ton to help develop it over the years. But RHEL is not merely the kernel nor any single upstream. RHEL is a product that comprises thousands of packages all tested together and then released as a supported product. > What Red Hat is trying to guard is not the source code to any single or even groups of projects. It's trying to preserve and capture the value it created from all those parts. Coincidentally, that value is what businesses, competitors, and the community are clamoring for and not the source code alone.

That's a point that (unfortunately) seems to have been lost along the way. A big part of the reason why people use(d) CentOS was because of the confidence they had that it'd be stable, functional, receive timely security updates, etc. And the main reason that was true is the work RedHat put into RHEL.

That said, it's not entirely a one-way street. The widespread use of CentOS meant much wider support for RHEL by open-source packages (and some closed source ones) than a locked down, limited availability RHEL would have had. I know of places that use RHEL because CentOS is widely available, and of places that would never bother to support RHEL if it weren't for the availability and (near) ubiquity of CentOS.

> What this comes down to is wilfully ignoring CentOS stream. All the clones of RHEL can simply track CentOS stream

You are willfully ignoring that CentOS Stream is upstream of RHEL. CentOS was downstream of RHEL.

Oracle Linux, Alma Linux, and Rocky Linux are all downstream of RHEL.

Feel free to argue all day about how similar CentOS Stream is to RHEL. Go ahead and argue how much it doesn't matter to you that it's now upstream.

The fact of the matter is whether it's upstream or downstream does matter to a lot of users and companies.

It does matter that IBM is trying to block access to open source code and is trying to restrict its subscribers from redistributing that same open sourced code.

IBM is causing extreme harm to the open source ecosystem that they profit from.

> The process of replicating RHEL is so much easier today than ever before.

I would argue that it was easier a few weeks ago when they made source rpms available publicly.

Reply on news.ycombinator.com