Dear Red Hat: Are you dumb? | Hacker News Reader

Dear Red Hat: Are you dumb?

jeffgeerling.com

702 pointsthepbone398 comments

Disclaimer: I'm a former Red Hatter but my opinion is my own

Red Hat, you NEED to fix this. I get that it sucks to have your business built on selling support, and a couple of companies have popped up selling cheaper support but not taking on the cost of building the thing. I get that (you think) it's going to cause problems for you (it might, I'm not sure yet). But you're torching an insane amount of good will that was built up over 20 years of good and mostly selfless deeds. You are making yourselves an enemy and a villain, rather than a friend. You need to stop the bleeding.

I don't have any answers, but I have some ideas:

1. Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support. If those don't agree, other projects will emerge who will agree and will take their place. Don't make their lives unnecessarily hard by throwing up roadblocks in their way. There will probably be popup companies selling third party support for Alma/Rocky etc here and there, but I doubt that type of support competition makes a serious dent to enterprise sales. In fact, I think it helps you by increasing adoption. A rising tide raises all boats, especially in this field.

2. Figure out a way to let individuals and small developers use real RHEL without having to dick with subscriptions or licensing or activation. Be liberal with this and make it easy

I think this is a key part of where they've been aiming for years. Since the '00s, even. Maybe earlier, IDK. I just have their public behavior to go on, but it sure looks like a long-term strategy. If it's not, it ought to be, because they're doing most of the work for it anyway.

1) Steer enough of the Linux ecosystem that Red Hat is the most-qualified support source for common Linux stacks, including maneuvering to make it difficult to decouple or avoid them in other distros (Systemd, Gnome, and Wayland are key plays here).

2) Cut off access to the blessed least-risky-for-enterprise version of your product.

3) You are Linux for much of the market. You own it. Profit.

At the (current) top levels, you might be right. I don't think it's been going on longer than 10 years though. I'd guess maybe 5 or 6.

The bulk of engineering doesn't like this, but sadly for a while now it's been a sales-driven company, and the IBM influence has certainly not helped. Sales doesn't give a shit about the health of Linux, or even the long-term health of Red Hat. They want their commission and promotion, and pesky open-source ideological things get in the way.

How about since July 9th, 2019?

What happened then?

That's when IBM bought RedHat.

I would say at least since udev was merged into systemd and gnome adding an hard dependecy on systemd-logind [2], both in 2012 so I would say more than 10 years.

To me that this was an act to force a Redhat-led project through other Redhat-led projects.

[1] https://lwn.net/Articles/490413/ [2] https://gitlab.gnome.org/GNOME/gnome-settings-daemon/-/commi...

Gnome and Wayland are the last worries for any enterprise customer.

Gnome's part of the play to make it painful for other distros to avoid Systemd, which does concern ~all enterprise customers. And I'm sure some of those enterprise customers are paying for graphical workstations, besides, even if it's a minority.

[EDIT] If I'm not communicating this well, consider: if Red Hat steers several key projects that basically all other serious distros are stuck with (in no small part due to Red Hat maneuvering that sure looks like it was designed to bring this situation about on purpose), how can another vendor ever be anything more than a knock-off of Red Hat? Ubuntu tried to fight them on each of these points, and others, and lost every time. Result: if you're running Ubuntu, much of your system is defined by, and the future development of it steered by, Red Hat. You're practically running a Red Hat knock-off, aside from their dead-man-walking FlatPack competitor, Snap. May as well just pay for support from a budget RHEL vendor... oh, wait, those just got cut off, guess you'll be paying Red Hat. It dramatically increases their moat and the cost of building a credible competitor that's not basically just off-brand Red Hat with less ability to steer the future of the platform.

systemd makes enterprise customer's lives easier. It's more reliable and it's easier to write your own units and timers than what came before.

Systemd's a lot more than an init system, and that's no accident. Red Hat now determines where Linux goes in some very important ways, and I think they believe they've finally got a solid lock on that, after years of effort, if they're making this move. Why go with a competitor (if you're a paying customer, which is who matters) that's just following in Red Hat's wake? And now, the options for RHEL-in-a-different-label will be gone. Who's left? Nobody, you're gonna pay Red Hat or eat extra (perceived, if not actual) risk.

[EDIT] Nb that for the above it doesn't even matter if Systemd's better than the pile of interchangeable daemons and programs it replaced. I have opinions on that, but it's irrelevant to Red Hat's market maneuvers, aside from that it was important to them that it not be easy & safe to cleanly replace, in part or whole.

Have you looked at 2022/23 commit logs for systemd and matched those authors to employers?

(Preview: It's mostly Microsoft, Meta and Suse.)

I think that's why this current issue is more relevant than ever. Red Hat is burning goodwill they had, which is going to eventually start to cut into the types of decisions they used to be able to make when they had more community "power". I've always enjoyed using Debian (or Debian-based) systems than Red Hat (or Red Hat-based) systems, and was unhappy when I had to re-learn init scripts, but at this point in time Systemd just seems to be easier with some features being "free" to restart or figure out systems that don't restart automatically.

The funny thing is that it looks like most of the decisions leading to that outcome have been overwhelmingly coincidental.

I would be very surprised if it's a coincidence that Systemd's as difficult to avoid, if you're building a Linux distro, as it now is, nor a coincidence that it's been going NIH all over the rest of the system ever since Gnome was leveraged to speed up its adoption.

The point is to make it difficult to offer a credible enterprise Linux distro that's not full of Red Hat-steered software in key places, turning you into a cheap knock-off of Red Hat with little say in what happens to the software you're providing, in terms of future development. It now costs a lot more money to get out of Red Hat's shadow & control, as a would-be Red Hat competitor—they're not just another distro among many, they run the damn show.

You missed 'deprecating the alternatives' :)

Somewhat jaded opinion after many years in the business, but these companies exploiting OSS for profit have learnt from how Microsoft's moves have played out over the years (and are now making their own mistakes). Given the only way they make money is by providing support, obviously they're going to go the MS route (make operation arcane and complex, as per your first point).

> You missed 'deprecating the alternatives'

I don't want to disagree too hard with this, because I, too, have been uncomfortable with the march toward making it very difficult to build a Linux system without things like systemd (I actually don't have all that much of a problem with systemd anymore, but I also value choice).

But no one company can deprecate anything. The source is all there, under permissive licenses. If people want sysvinit or openrc (etc.) to work well in order to build systems without systemd (for example), they are free to maintain, support, and evangelize them. And this happens, to some degree. If it's not happening to the degree we're all happy with, then we should pitch in to help, either with our labor, or financially. Otherwise we don't really get a say.

They just have to make it hard/risky/expensive to keep using things they've declared deprecated, or to avoid the software they're pushing & steering. Which they've done pretty damn effectively (again, if this were all somehow accidentally, holy shit, it shouldn't have been, because, wow, it's been a highly-successful-and-effective set of business maneuvers executed over a span of years)

Doesn't matter if some hobbyists still run Gentoo with openrc and X-Window with Windowmaker and all the classic interchangeable daemons and components that Systemd replaced (and with which it is incompatible)—it matters that any commercial competitor's only realist option, short of a massive and risky investment, is offering the same things Red Hat does, but with less influence on those projects. They want it to be expensive to build a competitor that's not a Red Hat knockoff—and if that's not what they wanted, well, somehow they took a whole bunch of steps over a span of years to make it happen anyway, by accident I suppose, but the outcome's the same.

> Doesn't matter if some hobbyists still run Gentoo with openrc and X-Window with Windowmaker and all the classic interchangeable daemons and components that Systemd replaced (and with which it is incompatible)

Don't forget alpine is also openrc and it is far more used by corporate tech than gentoo. Especially as a base for containers.

As ever, solidarity matters. The only way to get out of these embrace, extend, extinguish cycles is to form independent organizations that are never beholden to a single company or even a small handful. Individuals can't stand up to the efforts of teams of company-paid developers just by contributing their free time. But it's very hard to create such an organization without the companies immediately recognizing the need to stuff as many of their proxies into said org to gain influence, and that's a difficult thing to counter. As an ecosystem, we need to contend not only with whether a particular thing like systemd works well or solves a problem but also whether it has predatory intentions. This is simply what the incentives lead to unless the community has a strong voice to counter such attempts. And in this day and age, that also includes dealing with all the submarine articles and other consent manufacturing that are well within the budget of any large company to try to fake organic community discourse.

That sounds a lot more like Canonical's (attempted) strategies

1) Be the defacto "new user" or "newbie friendly" distro for two decades

2) Keep making asinine attempts to monetize your users (Amazon search bar comes to mind)

3) Try to pivot and break core Linux functionality so software only works on you (Keeping Upstart after everyone else abandoned it, Mir instead of Wayland and the horrendous amount of forking done to GTK and others to get that hideous 'Unity' desktop going)

Agreed, I think Canonical has been a lot worse in this respect than Red Hat. I think Red Hat's technical arguments have merit, and they do a pretty good job separating projects from the company so they are definitely not RPM distro-only.

Yes, Canonical has been their main competitor on each point and it's possible the threat of Canonical is what sent them this direction in the first place [EDIT: or vice-versa—Ubuntu may have attempted to "own" parts of their system in response to Red Hat's moves to do the same, I'm not sure how exactly that timeline works out and all I have to go on is how both have behaved].

Canonical-steered alternatives to Systemd, Gnome, and Wayland, have been advanced (as you note: Mir, Unity, Upstart) with less political and/or technical savvy than Red Hat's efforts, and have all failed. Technically they all still exist, in some form, but they're de facto dead. Snap/FlatPack is another place they're butting heads, and I think it's fair to say Snap's not got a bright future, but I guess it's technically still in the running, so we'll have to call that one undecided.

How exactly would Red Hat pursue such strategy and compete with Canonical? Consider the fact that Red Hat leaves the copyright to the contributor in Red Hat's projects, while Canonical requires contributors to sign CLA to get patches included in Canonical's projects. Given this arrangement, Canonical both benefits from the contributions and also has extra rights over the code that everyone else doesn't have that gives them a competitive advantage.

What's that got to do with maneuvering for control of extremely-important parts of Linux? If anything, license/attribution fuckery would put you at a disadvantage in that kind of a game. Red Hat was playing for position, not material, in chess terms. Canonical may have held more rights to their competing software, but it doesn't matter a bit, because those projects are irrelevant now. They "own" more, perhaps, but they're in checkmate, so it doesn't matter. Who's looking to pay the best-positioned claimant to "upstart maintainer" for support? Practically nobody, and that's not a growing market.

If you're an enterprise, do you see Canonical/Ubuntu or Red Hat as the most-credible vendor to support Systemd, which both of them are running? Is there anyone else who might hold that title? The answer to those questions are the point of what Red Hat's done (or else they've accidentally achieved a pretty amazing coup through a series of maneuvers that led to that outcome simply by chance, which notion strikes me as unlikely).

What's important is making it extremely expensive to launch or re-position a competitor distro to Red Hat that actually ships substantially-different software than Red Hat does, at a level of polish & reliability that might let you compete with them. It leaves competing distros as Red Hat followers. Which, fine if some hobbyists pick that, what they care about is being perceived as the only credible claimant to the title of first-party vendor of Linux, in most contexts by enterprise.

> I think it's fair to say Snap's not got a bright future, but I guess it's technically still in the running, so we'll have to call that one undecided.

I'm surprised they haven't thrown in the towel yet at this point.

When I read HN, I never see anything positive about snap. And while this is arguably a fringe community, it is very highly correlated with the 1% of influencers to corporate decision makers.

I think you can really only use this assumption if you are talking about desktop Linux. Using the Ubuntu server version is mostly giving you Debian with more packages available (and all the benefit and downsides to that).

> Keeping Upstart after everyone else abandoned it...

False. Ubuntu switched to systemd as soon as its upstream (Debian) did.

See https://www.markshuttleworth.com/archives/1316

Not false :)

Ubuntu offered Upstart starting in 6.10 (2006) through 15.04 (2015) after Debian's (admittedly very politicized) switch to systemd forced their hand

RHEL 6 (2010) included it for a single release before switching to systemd in RHEL 7 (2014) (though in balance, systemd was started by a Red Hat employee)

No other distros switched to it, and the ones that offered it as an option were very second-class citizens

So, they were one year after RHEL on switching to systemd to the default (and it was available as on option before that). The first LTS version of Ubuntu released after RHEL7 had systemd as the default. I wouldn't really say that they were using it "after everyone else abandoned it". I'd characterize it more as they developed an alternative init system that never really caught on.

Indeed. And notably, upstart came long before systemd, and with it Ubuntu was the first major (or any?) distro to solve the real problem of the non-parallelized boot speed issue for desktop users. Characterising it as "pivot and break core Linux functionality" is just outright misleading.

a.k.a. The more things change, the more they stay the same.

Few of us have careers long enough to remember purchasing SCO or HP-UX or similar and what that whole experience was like. We're positively blessed compared to how things were in the '90s.

Red Hat's leadership forgets what made them successful in the first place and wants those bad old days back badly.

When you're not at the top you want competition, since you are competition. When you're at the top you want to choke competition and gate as much as you can. It's a story as old as trade itself.

"Systemd, Gnome, and Wayland are key plays here"

GNOME and Wayland are utterly non-strategic for most of Red Hat's business. If your theory of Red Hat's long-term strategy hinges on those being strategic, then you need to go back to the drawing board. You could fairly cite systemd, I suppose, but Wayland and GNOME not so much.

On point 3 - that's not really controversial. All the Linux vendors have tried to "be" Linux in the public market. Canonical has certainly tried to be the brand-name Linux for most users.

To the extent Red Hat has tried to steer the market w/r/t its tech stack, it's done so pretty honestly and by trying to persuade others to adopt its tech stack and by doing the work in the open. This is the opposite approach of Canonical which has tried to push lots of NIH technologies to silo people to Ubuntu and continues to fail at it over and over again.

Gnome and Wayland countered Ubuntu's attempts to "own" competitor projects in those areas (the current fight is Snap/Flatpack). Gnome, in particular, was important as a lever used to make it expensive for distros to avoid Systemd, forcing it into de facto standard status more quickly than it may have achieved on its own merits (if it ever would have—and maybe it would!), from which point they've started using Systemd to eat other system services & daemons, increasing their control over the platform and cementing their status as, effectively, the "first party" vendor for Linux itself—not literally the kernel, but enough of the system that you've got to go pretty far into here be dragons territory, by the standards of enterprise, at least, not to be using Red Hat-steered software in key parts of your system—so if you want a say in the direction of Linux, as you experience it, or priority bugfixes from the best-possible source, or to talk to experts who write that software for a living, for advice or training, you're short on options other than... Red Hat. Which is the whole point.

Gnome is a healthy non-profit foundation with the majority of its organizational capacity (and income) not associated with Red Hat.

Red Hat hasn't been dominant in Wayland for years. It's very much a community effort, including governance and maintainership.

systemd is currently mostly developed by employees of Microsoft, Meta and Suse, aside from I guess Frantisek.

Red Hat does a lot, and I'm sure they have thoughts about where they spend their money, but I don't think people really understand how decoupled these projects and communities are.

It's funny how people view GNOME/systemd as some sort of conspiracy rather than it just being what the people who contribute want to work on.

Corporate strategy isn't conspiracy. I think it's even weirder to think redhat would just provide resources in return of nothing at all, or that redhat employees won't be biased towards steering the ship according to their employers best interest.

Like I genuinely don't think Gnome or systemd or Fedora would just use a ubuntu/canonical-built core system even if it was superior in every way

And hey I'm not complaining, I use systemd (indirectly) and gnome almost everyday. I don't care where the software comes from, really, I'm just glad I can use it. But that doesn't mean we can't be aware of the power dynamics around it :)

What about Upstart?

It was significantly better than sysvinit and Debian, Fedora, RHEL6, OpenSUSE all adopted it. Even though it came out of Canonical.

It was replaced by Systemd because that did concurrency in the dependency management, which Upstart didn't, and was never designed to do, so was going to require a lot of rework to support.

> the current fight is Snap/Flatpack

I work with enterprise customers every day who use RHEL, it's free variants, and Ubuntu - I can promise that none of them give a single shit about Flatpak or Snap. They will never use either of them. Those tools are only relevant to Linux on the desktop - the small size of that market is exactly why both companies (mostly Canonical) have gotten away with being a nuisance on this front.

Flatpak is very relevant on the immutable OS space which is being pushed hard as the future. It is mostly an enterprise (server/vm) thing too.

I just don't see people caring about an immutable OS in the enterprise space. The central requirement is already addressed by containers, and all of the security and compliance stuff enterprises do is predicated on having a "normal" OS - even if an immutable deployment is technically more secure, that doesn't help you if your auditors can't understand whether it checks the requisite boxes.

Yeah. To me, this just seems like IBM feeling like their existing (corporate) sales funnel is big enough that the consequence of this (reducing OSS users based sales funnel) aren't going to meaningfully matter.

Or, at least they won't matter while the current decision makers are still at IBM. It could be a problem for future IBM leaders, who will probably just attempt to buy SuSE or Canonical or something at that point. ;)

GNOME was pretty instrumental in forcing systemd to becoming a GNU/Linux-wide standard though, by creating some dependencies there.

> This is the opposite approach of Canonical which has tried to push lots of NIH technologies to silo people to Ubuntu and continues to fail at it over and over again.

Agreed, Canonical is worse. The whole closed snap store (where only canonical can use it) is an obvious play on milking the commercial software market.

That doesn't mean that what RH is doing is great though.

> Systemd, Gnome, and Wayland are key plays here

Systemd, sure, but I would assume that the vast majority of Red Hat's support contracts would be for server use, where GNOME and Wayland are irrelevant.

Gnome was a major lever in making it hard for other distros to avoid systemd, early on. The way Wayland's been structured and that... transition, such as it is, has been managed, smells a whole lot like a fire-and-motion move, and besides, dominance there was necessary to keep Mir from winning as the successor to X-Window and making it possible for Ubuntu to follow Red Hat's playbook and use it to push some of their other, deeper-in-the-stack projects (they'd have been neatly positioned to interfere with both Systemd and Gnome, had they won). I don't think Red Hat would let Ubuntu push anything of note, at this point, without countering it, for (justified!) fear of their own tactics being turned on them.

https://www.joelonsoftware.com/2002/01/06/fire-and-motion/

Systemd's the main body of Red Hat's army here, if you will—no doubt that's the most important part of all this—but other projects have played other important roles in supporting or defending it, at times.

It's not like Red Hat planned and schemed to execute on Wayland as a plan for domination, but their decision to support Wayland as an existing project with technical merit staffed by skilled people might well have been tactical for the reason you mention.

Some people tend to imagine the first situation and rightly dismiss it as a post facto conspiration but the second situation is utterly realistic. They may even not have realized until much later how right they got.

And I say that as someone who is convinced we need something like Wayland, the sooner the better. But we also have to avoid being naive about how business is done.

>You are Linux for much of the market. You own it. Profit.

What a twist!

So after all it wasn't Microsoft EEEing Linux, but actual "Linux Companies"?!

Microsoft tried and failed because their product is strictly inferior for the market which Linux has ended up dominating. Marketing and evil behaviour can definitely win out over technical quality, but we were fortunate that it didn't go that way in this case.

I'm not sure if that argument holds water. You have Azure which is hugely successful, and Linux is still not anywhere close to becoming a standard desktop environment beyond the type of people you find on HN. Microsoft contributes quite a bit to Linux now, because they see the future. Becoming a leader in open source gives you the ability to steer where technology goes, and Red Hat definitely was able to play that hand as well (but under IBM seems to be losing quickly).

I'm not sure that any of your post actually contradicts mine; MS have accepted the dominance of Linux in the server market and have moved on to exploiting it where they can. That doesn't mean they wouldn't have rather had everyone running on Windows, if they could have pulled it off.

Maybe a solution for point 2 could be that the first five licenses or activations are free.

That would help, but I think that only works optimally if they also do number 1. If there is any "activation" process or the like, it adds enough friction to cause problems, especially in a modern CI/CD environment. It's not an easy change by any stretch, but if they don't do something they're going to be the next Oracle.

I worked for a company that did the voting for American Idol for the first few seasons. Weird culture, a mix of bare metal and NIH that I thought didn’t quite function and some of us were trying to change that culture. Everyone resented working on that project and wanted to do other things, so at some point we disinvited ourselves (or so the story went, might have been sour grapes).

The problem was we killed the cash cow without any viable livestock lined up to replace it, and the layoffs started about a year later. Of course 4/5ths of the people trying to change the culture were in the first round. Get rid of the rabble rousers first.

While I think our proposed changes would have helped dramatically, I also think it might have been better if the company management had invested in making it more palatable to work on the project. Bribery if you have to. Conferences, toys… but they didn’t do either, and it flopped.

Edit: on rereading your comments I may have missed your angle, but I’ll leave this anyway, since we are talking dumb failure modes.

That I can’t run sudo yum update out of the box with RHEL is one of the big reasons I don’t bother with it. I love the “stability aspect” of RHEL but I love the I guess “convenience” of any other distro being ready to go out of the box, no activation required

I totally agree. Even when I worked at Red Hat and could get whatever licenses I needed, I still used CentOS for local stuff and for testing things because of the friction of the registration/activation process. It has gotten easier in recent years, but simply existing is enough to turn off a lot of people.

"Me too" - Ex Red Hat. Development is done on Fedora and CentOS. I was pissed when they changed CentOS mainly because it changed my workflow.

Since RHEL 8.2 its at least possible to register at installation time.

Alma, Rocky, fair enough.

But isn't the elephant in the room here Oracle Linux?!

Yes, very fair point. Oracle and Amazon both deserve to be on the list, probably much further up than Alma or Rocky

Amazon actually seems to have seen this coming. While Amazon Linux 1&2 was a basterdized CentOS6& derivative (with some newer packages), the latest breed (Amazon Linux 2022/2023) is based off Fedora.

Also, I'm not convinced Amazon (or Oracle for that matter) have a very bright future. The opaque development process and (lack of) community support make development pretty hard. I know from experience, getting bugs resolved (or just getting Amazon to care) is very hard. At the same time, even for a large corporate maintaining your own distro brings with it considerable costs.

Wouldn't it be great if instead they would work together, share some of the costs and profits of support and cloud mutually?!

Amazon missed then, the company to bet on is SuSE here.

Both Oracle and Amazon Linux have VERY bright futures for their use-cases.

Do you run Oracle? Want to know your distro will be supported 100%. Oracle Linux seems like a damn good bet to use.

Are you one of the largest companies on earth, who needs a distro to run your business? If so... guess what.. your distro is viable just on your own business.

Amazon offering Amazon Linux is just them showing some internal tech. Nothing more.

Amazon (and Microsoft) offering a Linux distribution is about (a) raining in the explosion of distros their various teams depend on that they don't themselves control, and (b) answering insistent cloud customers requests for "What distro do you recommend? Do you support {distro}? Do you auto patch {distro}? How come update to {distro} broke your service/product? Do you not work with {distro}?

Last year many services on Azure went down because they all used Ubuntu and Ubuntu pushed an update that broke for a DNS configuration that Microsoft uses. Having your own distro won't guard you from fuck-ups of course, but you can imagine the difficult conversation Microsoft had to have with their biggest customers about how Canonical can take down your 40 billion dollar business globally?

> Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support.

That was CentOS (after acquisition by Red Hat but before CentOS Stream). Unfortunately killed off.

Agreed, although there is one important difference: Red Hat isn't behind Alma or Rocky like they were CentOS (yes they did that to themselves by acquiring CentOS, but that was in the past. It was a mistake, but we have to move forward from where we are today). A lot of people knew that CentOS was owned by Red Hat and that made them a lot more comfortable using it. They kind of had to change that relationship somehow. I'm not happy at all that they did it, but I can understand why.

Option 1 above allows them to keep that "those are third parties, not Red Hat" relationship but still have the benefits of a free community build.

International Business Machines¹ should merge with the International Brotherhood of Magicians² to get a better chance of pulling this off.

[1] https://en.wikipedia.org/wiki/IBM [2] https://en.wikipedia.org/wiki/International_Brotherhood_of_M...

> Red Hat, you NEED to fix this.

No, they don't. They don't need to do anything.

> But you're torching an insane amount of good will that was built up over 20 years of good and mostly selfless deeds.

Doesn't matter. That goodwill doesn't really benefit them. How many companies paid Red Hat because they liked the things they did for open-source? Realistically, very few if any.

> Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support.

Realistically, there are lots of companies that would pay for Red Hat but don't because they get can get it for free.

> Realistically, there are lots of companies that would pay for Red Hat but don't because they get can get it for free.

If the free RHEL variants disappeared today I'd wager most of those companies would choose other distributions (Debian, Ubuntu, openSUSE, etc) or making the jump to containerisation or serverless before paying for RHEL. In the end less people would be using their kind of Enterprise Linux, which in turn will result in less conversion to the paid version of it.

> In the end less people would be using their kind of Enterprise Linux, which in turn will result in less conversion to the paid version of it.

Why would anyone want to convert to the paid version if their use case is covered by the free version? Out of generosity?

Sometimes support - for the OS or from applications - is actually useful. I worked at a shop once that ran CentOS for most things but RHEL for the database servers, because the vendor only officially supported that.

Additionally as their needs grows and the free version doesn't meet their needs out of the box and the non-free version does they'll be more apt to buy into it given their previous experience with their products.

So the reason for not using RHEL everywhere is the pricing model? I wonder whether it'd be possible for RH to optimize it so that choosing RHEL 'd still be the best option for them now that CentOS doesn't provide the same level of stability.

> I wonder whether it'd be possible for RH to optimize it so that choosing RHEL 'd still be the best option for them now that CentOS doesn't provide the same level of stability.

What you may not be aware of, or have forgotten, is the free versions provided by other companies that provide package compatibility and similar support (as in updates and security fixes) that CentOS did.

Oracle Linux, Alma Linux, Rocky Linux, and Amazon Linux. Quite possibly even more.

Red Hat will have a hard time undercutting them just by reducing their pricing, which is why they're using tactics like a free number of subscriptions/licenses for small organisations and developers and trying to cut off access to the source code.

In our use-case, needing to handle licensing in any form for additional nodes we scale up in non-production would break our processes. We run RHEL on some core infrastructure with equivalents in our pre-production but CentOS everywhere else to give a ‘close enough but free’ infinitely scalable environment to our devs. As the person in charge of the existing relationship with RHEL, there is no amount of discounting or special deals that would make the extra complexity of registration worth it. We need an unencumbered freely deployable OS for our non-production, and for us the result of this change is likely to be a re-platforming to something like Amazon Linux and a seperate support contract or line item on our AWS relationship, rather than giving RedHat any more money and time.

For us, this means we stop doing business with RedHat.

There are plenty of cases where infrastructure can be split into "critical" and "nice to have".

In HPC, it is reasonably common for people to run RHEL on the scheduler, database and login nodes (totalling maybe a dozen servers) but then use CentOS (now Rocky) for the compute nodes, which might be hundreds of nodes. Those customers simply can't justify the cost of RHEL everywhere, even with more recent changes which make it cheaper than it has been historically in this use-case.

To me it sounds like the pricing model is not optimal. I guess in these situations, they could pay RH "a bit more" in exchange for a legal RHEL license covering those extra nodes. But if there was no such offer then it sounds like a lost opportunity.

(but I'm a complete outsider, I just share my views and I do acknowledge I may be missing something, so feel free to elaborate further)

I think you are right - the problem is that "a bit more" really ought to be a small uplift on the needed licenses, rather than the current inadequate discount on compute nodes (along with hoop-jumping to convince sales people you aren't trying to scam them).

Boeing, up to at least 2016. A large amount of RHEL, and a screamingly HUGE amount of CentOS.

It is usually some combination of needing paid support if updates break things, and also generosity. Trust and generosity define the choice of paid support organization.

I had a meeting with our RH reps last week, as a followup to Red Hat Summit and to talk about a potential purchase with them. I had a discussion about how we use free RHEL derivatives, and have for almost 30 years (when Red Hat itself was free), and we might have used one of the first public releases (I do know we were running 5.2 in '99).

In explaining our position I told them we use the free products (which meant CentOS until stream, and now Rocky), I noted how I'm sure they have their own reasons for what they did, but the Stream change made us seriously consider other distros like Ubuntu, and while we settled on Rocky in the end after it came out, if we had chosen Ubuntu we wouldn't be having a discussion about possibly spending $100k with them now and more in the future, so that's how they're hurting themselves.

Then again, from attending Red Hat Summit, it's apparent that they're really aiming at the enterprise and government anyway. I would estimate attendance was 60% government and education (including a lot of government contractors), 30% enterprise (Adobe, etc), and 10% small to medium business (and I'd bet many of them are partners offering services and want to support Red Hat).

To me it looks like people making short sighted moved to increase their hold on their enterprise customers (this might be aimed more at Oracle Linux than Rocky or Alma), but at the expense of the surrounding ecosystem. The knock-on effects might be that they kill the ramp to RHEL and the training of admins in its use except for those companies willing to actually shell out for official training programs, as there will be little to no regular use by non-enterprise customers anymore if they keep this up.

>> there will be little to no regular use by non-enterprise customers anymore if they keep this up.

They are slowing transforming RHEL to become the new AIX (https://en.wikipedia.org/wiki/IBM_AIX).

What is that line? “No one ever got fired for buying IBM”?

I think the main appeal for enterprises is that if their VM’s start going down and thousands flush through the drain every second, someone at Red Hat is there to pick up the phone and hop on a call with their engineers. It makes them and their regulators happy and no other Linux distro offers that. I don’t think they care about open source at all.

> What is that line? “No one ever got fired for buying IBM”?

It's a wildly common saying in enterprise, and it basically means, "if you choose a startup or unknown product, it might go really well or it might be a bomb that gets you fired. If you choose IBM, you'll spend more, get less of a product with less quality, but you'll never get fired for it because IBM is a 'safe choice'"

> I think the main appeal for enterprises is that if their VM’s start going down and thousands flush through the drain every second, someone at Red Hat is there to pick up the phone and hop on a call with their engineers. It makes them and their regulators happy and no other Linux distro offers that. I don’t think they care about open source at all.

Correct, enterprises don't care at all about open source. This is IMHO a bit part of why Red Hat is stopping caring as much.

Now that said, when I was a consultant with Red Hat I worked with a ton of RH customers and there were individual engineers who did care. Not nearly as many as I hoped, but there were at least some.

Also former Red Hatter, disclaimer also applies.

According to my observations, Red Hat does an unbelievable amount of stuff by hand. Lots of internal processes require someone to do something at a given point, otherwise these distros are not happening.

I believe RH is now cutting costs at precisely these places: projects that don't directly contribute to the revenue get cut. However, as the post said, these changes make it incredibly hard to ship open source software for RHEL-likes to the point where I already decided to look for alternatives when doing integration work.

I would like to add the perspective of a small-business infrastructure guy here.

First things first: we're a Rocky Linux shop. About 100 instances, some bare metal, some VMs. A few local, a few at cloud providers like DigitalOcean. Held together by Ansible and kickstart installs.

Too often the discussion about RHEL clones like Rocky paints a picture of people like me as freeloaders. And some of the folks with a similar setup might be. But most small-time Linux admins I know are real open-source believers. I would really like to funnel some money into the platform we're using. First and foremost because it's the right thing to do, the people writing all that source code deserve it. But also because we need platform continuity, and being able to open a support case from time to time would also be great.

But it's really hard. I don't have the time and nerves to deal with the RHEL sales department, deal with the licensing, produce my own cloud images because DigitalOcean can't provide them, deal with our own internal purchasing process, deal with convincing my superiors that we need to.

Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

We're the same, but a little larger. Don't let anyone convince you you're a freeloader. If you're a freeloader, then Red Hat are freeloaders for relying on the Linux Kernel and all the open source software in their distros as well.

What we both are are power users that don't really need their support. Our company has an entitlement so we can use the KB articles (to our benefit) and view/submit bug reports to them (which is to their benefit). I have no problems telling them that (and for the most part did last week), and if someone on the call were to argue me on that point, I'd have happily gotten into it with them. I know they're adding a lot to linux and open source, and pay for a lot of developers to maintain and add to projects (including Linux itself), but that's just the point, they're adding to something that already exists that they use. They don't own it, and they can't expect to require payment for it. They aren't Microsoft, and the only reason they've ever been able to compete in that space is because they're standing on the shoulders of what the open source community has made possible.

It's a symbiotic relationship, and they can mess with it at their peril.

> then Red Hat are freeloaders for relying on the Linux Kernel

"freeloaders"

Most Active Linux 6.1 Employers [1]

By Changeset

[...]

7. Red Hat 672 4.8%

By Lines Changed

[...]

10. Red Hat 24073 3.1%

Most active employers, 5.16 through 6.1

By changesets

3. Red Hat 4916 5.7% (behind Google and "Unknown")

By lines changed

8. Red Hat 202698 3.3%

[1] https://lwn.net/Articles/915435/

and that's just the kernel. Red Hat also runs or contributes to a bunch of other projects that are significant, especially relating to containers.

Not just containers, all the core linux stuff. gcc, binutils, gnome, systemd, etc.

One of the major differences between buying a support contract with RH and buying one from literally any other vendor is that RH doesn't sell support for things without having someone active in that project's community.

AKA, there is a good chance the gatekeeper that your !RH support provider is talking to is a RH employee or funded by RH. Put another way, rocky can't afford to hire 1k engineers to maintain all those packages, nor can anyone else so your trusting that the bug you want fixed can be fixed by a drive by contributor. This is part of the reason that RHEL has such a small package list too, just about every single package in that list has someone working upstream.

So, you won't get stuck like i did 15 years ago with a glibc bug that the company your paying a support contract to can't fix. Sure that distro patched their own version, but the change got dropped a couple years later during a rebase, and it was the only distro with the fix. Meaning our product only ran on that specific distro + version until we reopened the bug with RH and they actually got it fixed upstream.

To be clear, you're less likely to get stuck like that. There's plenty of bugs open that Red Hat doesn't fix because it's either too problematic or affects too few people. Sometimes that's eventually solved with a point release and them rebasing to a newer package version which includes an upstream fix, sometimes it just persists. I'm not saying it's common, but it's happened, and happened to us. Encountering a bug and finding the KB article with only workarounds and then a bugzilla entry that's been open for a couple years isn't exactly common, but I also wouldn't count it as all that rare in my experience.

Also to be clear, I wasn't making a case abovc that Red Hat doesn't provide a lot of benefit to the open source community, just that Red Hat could not exist without that same community. I do not begrudge Red Hat making money. I do think they're being extremely shortsighted if and when they shut off the larger open source community's uses of their core OS product. They're making their whole ecosystem much harder to get talent for if they do that, and it's not like entities running Rocky or Alma are just going to spring for RHEL. We trade on our staff's skill as System/Linux Administrators, and the core OS is not as important as all that. It's like if Python started requiring licensing fees. You'd get some short term return on that and some larger entities will just pay, but you'd see python usage in the general community plummet, with all that entails about its future.

Maybe to their eyes it makes more sense in this current world of startups and VC to just exclude the smaller players, since those companies that grow organically to a size where Red Hat's pricing makes sense are much rarer?

>Not just containers, all the core linux stuff. gcc, binutils, gnome, systemd, etc.

Depending on who you're talking to, this may not be a good thing.

... and looks like they're locking many of those changes to the source code behind their own paywall.

While it may technically be legal (ie crafted by lawyers to make sure of it), that still seems like a pretty shitty thing to do.

Red hat upstreams a lot of their patches, much more than Debian or Ubuntu.

I'm aware, I used to work for Red Hat (2010-2015). :)

There's also different types of contributions. Hardware vendors like Intel and AMD are mostly contributing their own hardware drivers (and a lot of that, especially in AMD's case, is just autogenerated hardware definitions), whereas Red Hat is mostly contributing to core kernel functionality.

> If you're a freeloader...

It's dishonest to leave that conditional off the quote. The work and risk/stress involved in running a supportless shop is quite high. So if they are freeloading by using Rocky then Red Hat is also freeloading.

I do think it's funny that they weren't even number 1 on that list though. Guess they are at least getting a lot of volunteer work from other companies!

The others are almost all hardware manufacturers doing new hardware enablement.

It's hard to tell without more context of your own whether you're supporting what I said, or refuting the argument that Red Hat are freeloaders, which is an argument I did not make.

Thanks for the feedback!

We've just about finished transitioning to 100% Rocky, coming from a 50/50 setup with many of the bare metal installs being Ubuntu. But Ubuntu left a real bitter taste in my mouth with the snap enforcement (I can't even start this snap thing in my Ubuntu Docker container? And you took away the native apt? Wow.), so we started a migration to 100% Rocky.

Telling my manager that we'll have to work something out after we just about finished working something out will be my own peril as well. Let's see where this will end.

I would wait a bit to see how this shakes out before raising the alarm. I'm waiting to see what Alma/Rocky do, as well as holding out a small amount of hope that RH and others will agree to something.

Yeah, we were late on CentOS 8, and had just put 3-4 systems and VMs into production when the Stream announcement hit, which made us stop in our tracks. We eventually accepted Rocky and started actually getting newer systems in place once 9 was out, so we just jumped to using 9 and in-place upgraded our Rocky 8 systems to 9.

We're only about a quarter migrated though, so still have a few hundred Centos 7 boxes to eventually get off of, and we'll see if that ends up being Rocky or something else.

I will note that this episode of The FOSS Pod[1] with a Debian lead actually made me think switching to Debian itself way back in the past (since we've been doing this for almost three decades now) would have been a safer path. I highly recommend it, it cleared up a few misconceptions I had as a Red Hat/CentOS user about what Debian is really attempting and why, and how that would fit with our needs. For example why their goal of pure open source licenses isn't an ideological choice, but a choice to benefit the Debian ecosystem, since the goal is to make downstream distros of Debian easy and without problem (e.g. no extra legal review needed based on licensed included).

1: https://fosspod.content.town/episodes/debian-linux-with-jona...

Same here, we still have 2 or 3 CentOS 8 to Rocky conversions to go from before we even started replacing Ubuntu!

I have to admit, while I share a lot of the values of Debian, I never really liked the product. Especially the apt package mangler, err, manager. They've also carried some Debian in-house patches that really caused me some grief in the past.

I've started running FreeBSD on my personal servers. Unfortunately it's not an option at work.

I've never really used Debian in the past, so can't comment on the reality, just what I've heard, and even my Ubuntu experience is lacking. I'm running it as a virtual desktop now, but I try to keep my use of it to an MUA, web browser and terminals so I don't have to spend the time to become familiar with it, given it's got no other use for me so far.

Our director loves FreeBSD, and we actually use it for firewalls, so it's not a spectacularly hard sell. I'm not sure it's significantly different from the problems of Stream to make it worthwhile, at least not without some external patching and support entity in addition, but it's been a long time since I looked at the specifics of their packaging and ports systems. I don't actually mind something like Stream for a personal server actually. It's just when trying to manage a fleet in the tens and hundreds when it becomes a real liability and increased possible admin support load IMO.

Asking from curiosity, what is the problem you had with apt? I'm using Debian close to 20 years now, and liked Debian and .deb a lot, more then RPM even.

Honestly, having built some packages and maintaining a derivative provided me more insight than a regular user, but even before that I was using Debian without any problems.

BTW, Ubuntu is not Debian with a different UI. They're pretty different on fundamental things, even.

A broken pre/post install script (often the problem is not expecting the current state of your machine) that fails while running `apt-get install -f` it the common culprit.

Just curious, why didn't you migrate from Ubuntu to Debian? Wouldn't that have been a smaller change?

Because we already had a solid base of RockyLinux VMs, so standardizing made sense. And I personally dislike apt / Debian.

Did you really compare our freeloading shop against one of the biggest software contributors to Linux and supporting ecosystem? RH is a top 5 contributor of patches across the most important core components.

Red Hat adds to the open source projects, they didn't write them from scratch (in most cases). Using freely provided software and building your business on it is perfectly acceptable, whether it's us using Red Hat's sources or Red Hat using the Linux Kernel sources and hundreds to thousands of other open source projects. We extend the OS where it doesn't fit our needs, and submit patches and feedback and bugs to both Red Hat through a subscription for a system we have for just that purpose, as well as to the upstream projects RHEL is based on, just as Red Hat contributes to the upstream projects.

So yes, I made that comparison. If you think a company using a RHEL derivative product is freeloading, I'm not sure how you can believe Red Hat isn't also freeloading on the open source community, but feel free to explain you position.

I, for one, think neither is freeloading, but just making use of the open source community to provide extra value for ourselves and our customers.

> Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

I think they do this on AWS and Azure (and maybe GCE?). DO may just not see enough of their enterprise customers asking for it for them to dispatch the lawyers and negotiators to make this happen.

Disclosure: Ex-Red Hatter, but never got anywhere close to anything related to sales or licensing.

Thanks for the hint! I will keep it in mind. Still waiting for the situation to really calm down. We'd rather not spend precious time and effort on changing clouds, and even if we did we'd still need to figure out the situation regarding bare metal and container images.

Most likely switching from DO to AWS or Azure is going to greatly increase your costs

>Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

We've tried to make deals with RH in the past, but they will only let us sell RHEL VMs with licenses if it's running on a RHEL hypervisor... which none of our platform is using at all.

I think this sucks, but for a little bit of defense of RH on this the entire support stack is built around assuming a RHEL host (meaning you're using a supported and exact version of the hypervisor and associated tools). Adding support capabilities for a non-RHEL host would be a major task, and not one they are well equipped to do.

Now that said it's a bit dumb because with most things it's not hard for support to tell your customer, "yep that's a problem in the guest" and help, or "that's a problem with the host, contact DO" or whoever.

But in general, when Red Hat says something is "supported" they are guaranteeing a lot about it. They'll literally fix the code if something is wrong. There's also a culture internally of "be careful helping with something that is 'unsupported'" because customers have previously accused RH people of messing stuff up and demanding they fix it. I helped with stuff like that most of the time when I worked for RH, but I was always very clear with the customer that "this isn't Red Hat, this is me helping as a friend and fellow Linux nerd" and the rapport with the customers was always good enough that I wasn't worried.

As someone who has done this, and benefitted from others doing this, I think the open sharing is probably the number one reason to use these types of solutions. You get people that are in their own niche and know specific parts of the system inside and out. That builds credibility and increased profits, but the companies are too nervous to stand behind their people.

> you're using a supported and exact version of the hypervisor and associated tools

Having exact versions is not a problem (any version of RHEL guests are supported on any RHEL hosts past and future, almost). Plus Windows, ESX, Amazon and Google clouds, and a few more.

Rather, the problem is having _up to date_ versions; with RHEL you can often ask the customer to reproduce on an up to date host if possible, with other distros the support machinery isn't there and you need someone to talk to at the cloud provider, so that they can debug stuff that breaks only in their environment and can't be reproduced. Otherwise you cannot guarantee the level of support that you mention. Very few cloud provider can provide that, I am not even sure that IBM's cloud made the short list.

Isn't long term support Red Hat's thing? The ask isn't to support some random system, it's for RH to put out an official image and distro they support on cloud providers and bake the licensing costs into the hourly runtime fees.

This is exactly how you consume Windows on AWS and it's a huge incentive to do it that way because it takes licences management out of consideration.

The cloud provider is also running software, the cloud doesn't run on magic dust. Red Hat doesn't supports RHEL on your blood if they don't believe they can trust you to help supporting RHEL, the way their customers expect. The easiest way is to run RHEL as the bare metal OS but it's not the only way.

> This is exactly how you consume Windows on AWS and it's a huge incentive to do it that way because it takes licences management out of consideration.

It's also how you consume RHEL on AWS, GCE or Azure.

Blood -> cloud of course.

That's sad.

And we'd have to figure out what to do with the bare metal instances and containers, too.

"Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing."

This is a two-way street. It's not all on Red Hat. I don't know to what extent Red Hat has or hasn't tried to do a deal with Digital Ocean but it's not as simple as Red Hat just deciding to do it. There's a fair amount of certification work that DO would have to do as well as hammering out the legal / business agreements.

The "comparable" part exists with many cloud providers, so if you're not married to Digital Ocean then that part should be doable. Red Hat also has some programs around subscription portability.

However, I do get it that the subscription process introduces friction and for small shops that have a patchwork infrastructure as you describe, it's going to be difficult. Some of that is on Red Hat to solve if they want your money, but not all of it. They can't do it all themselves by fiat if Digital Ocean isn't willing to meet them halfway.

We're not married to DigitalOcean, but changing cloud providers also takes time and investment we'd rather spend elsewhere. And we'd still have to figure out what to do with the bare metal instances, container images and whatnot. The friction is very real.

you have to understand, why would digital ocean want to deal with this when they can use actual open source options without the overhead?

Because they might some day find themselves in a tight spot if all of those open source options have become a half-broken wasteland because nobody invests time/money.

being afraid of all free linux distros going away because no one invests in open source is some epic level FUD.

How exactly do you think those companies even came to exist again?

From your description, IMHO you're not a freeloader, especially since you want to contribute. You are the exact type of people that I think Red Hat is harming themselves with.

I'm in basically the same boat, currently using Linode. At other companies I've made decisions to buy RHEL and OpenShift, and would again because I've been pretty happy (we'll see how this current thing shakes out).

I also went to work for Red Hat (and took a pay and title cut) largely because I had so much love for them. I'd been using Fedora and CentOS (pre-stream) for many years. The expertise that I gained from the experience also let me hit the ground running in my new job, and I made them a lot of money. Had CentOS not existed, I would never have learned their ecosystem and certainly wouldn't have gone to work there. That's profit that is hard to measure, and it is hard to directly tie it to somebody's commission or KPIs, but I'm sure it's not unsubstantial. Many of the people I worked with had a similar backstory!

I would also say (at least when I worked there) I didn't know anybody (who wasn't a sales person) who would have called us free loaders. Quite the opposite, they considered us part of the community.

This is essentially my backstory as well. I'm typing this on a Thinkpad running Fedora. RedHat seems to be focused on big corp and government customers. But even they are not an island. Loosing the grassroots will really hurt them in the long term I think.

Honestly, there are a ton of people like you out there. A big selling point for Red Hat has always been their contributions to the community and that funding them would help to drive that.

It is a big motivator for people transitioning from your state to becoming a customer.

> Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

Most of the major clouds have this pre-existing capability to charge more and revenue share the extra through their marketplace services etc; and they're in active use by other vendors. It's often a good source of revenue for both the cloud company and the vendor, and helps the customers by providing them with prebuild solutions that they don't have to build themselves (e.g. web application firewall vendors can/do provide closed source images that customers can use that way). It's not like this is expecting cloud companies to build whole new services from whole cloth just to support Red Hat.

>Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

This 100% already exists, but I don't believe Digital Ocean is one of the supported vendors.

You with 100 instances is pretty sizeable. On the other hand, I know research infrastructures (read: thousands of instances around Europe) built with CentOS 7, Alma & Rocky.

These guys are also power users and developers who need no support.

This is a massive blow by RedHat, and I'm not sure that they're aware of the size of the install base they are attacking to.

> Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

This exists, but probably only for the larger cloud providers and not e.g., DigitalOcean: https://azuremarketplace.microsoft.com/en-GB/marketplace/app...

> Red Hat, you NEED to fix this

You should be addressing IBM. This is how they operate. Things will only get more strict and more difficult to work with from here. They don't care about good will. They don't care about OSS. The only thing that matters is dominance in the market place to take care of their bottom line.

A lot of Red Hat people pay attention to forums like this, and I would bet money there's a big discussion going on internally right now. IBM couldn't care less, but the people at Red Hat will. They're also not powerless (at least, they weren't when I left a few years ago).

There were similar discussions when the CentOS Stream thing was announced and Red Hat didn't change course so I'd guess they're not going to back off this change either.

> There were similar discussions when the CentOS Stream thing was announced and Red Hat didn't change course so I'd guess they're not going to back off this change either.

True, and it's likely nothing will change this time too. But if there's no outrage at all and nobody tries, we guarantee nothing changes and probably invite further encroachment.

Obviously IBM owns Red Hat now and all that. But people overestimate how much involvement IBM has day-to-day with Red Hat. (Source: I'm a former Red Hatter, left last year.)

At worst I'd imagine that IBM's influence is felt this way: IBM expects x% growth and other metrics every quarter. How that is achieved is up to Red Hat's executives. If the leadership at Red Hat decides it can do that with the status quo, then I don't think IBM is going to say "no, you must change the source distribution to fit our evil plan."

The worst thing that IBM has done directly (IMO) was plucking Jim Whitehurst away from Red Hat and over to IBM, but not making him CEO. Whatever reason IBM chose to do that, it wasn't good for Red Hat or IBM long-term in my opinion.

almost 20 years ago when I took a Red Hat Kernel internals class (at Redhat) they basically told us to use CentOS if we wanted to experiment with RHEL. (We were preparing for a transition from the HPUX/Solaris machines.). But it worked, some of our architects did that and we could get the scheduler behaving similar to HPUX. I left that company, but I'm pretty sure they did switch and RedHat got a big contract out of it.

We've come a long way.

> 1. Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support.

I don't know how the support business breaks down for this, but for the large customers, I can't imagine any of the little rebranding projects taking the support business that the IT dept. itself calls. "Nobody ever got fired for buying IBM," and in this case it's literally true again: RH still have some of the best technical experts.

(Imagine being a huge company with a urgent mission-critical kernel problem: you want to be able to get a Red Hat commando team parachuting in, including a Sr. Principal Engineer who specializes in hard debugging tasks in that vast code base. Not have Bob's Discount Distro's support line be lost after you say you already tried rebooting and reinstalling your global operations.)

At the same time, if anyone distro is rebranding RHEL not just as an open source hippie volunteer thing (that a lot of not-ready-to-pay-for-RHEL smaller businesses use), but also trying to make it a support business out of it, then that might be poking the lion. The hippie commune might get ripped apart in a rampage. Which might be what's happening now?

(Credible support options from AWS and maybe Oracle are a different matter.)

> 2. Figure out a way to let individuals and small developers use real RHEL without having to dick with subscriptions or licensing or activation. Be liberal with this and make it easy

Maybe CentOS had been providing this market segmentation with upgrade path to enterprise support, until recently? RHEL is not easy to switch to, and I've had a client that was very capable shop look at switching to RHEL, and nope right back, because they didn't want to learn the new bureaucracy. If they'd been able to use RHEL for free since they were smaller, becoming an RH enterprise support customer as they grew would be easy and a no-brainer.

(Docker and then K8s have shaken this up a bit.)

>I don't know how the support business breaks down for this, but for the large customers, I can't imagine any of the little rebranding projects taking the support business that the IT dept. itself calls. "Nobody ever got fired for buying IBM," and in this case it's literally true again: RH still have some of the best technical experts.

I support a very large telco company, and they've had a massive effort underway to convert all of their RHEL systems to Rocky (preferred) or Ubuntu because of the loss of CentOS as something they could use on non-production systems plus their (the telcos) financial issues making them look for any software licenses they can abandon. They've replaced thousands of RHEL boxes with Rocky boxes already, all public cloud images must use Rocky or Ubuntu, no RHEL allowed.

The use of RHEL is limited to critical systems like those running an Oracle DB only (and even those they are investigation replacements for).

why even go through all of this when debian is sitting right there, free and reliable.

Not sure if Red Hat will be able to fix this. This is not Red Hat but IBM. If IBM could fix problems like this, they wouldn't have been declining in the past decade.

It's not because they're not dumb, it's because they're IBM...

Y'all are talking to the subsidiary of a Fortune 500 like it has ears, or feelings. It doesn't. It doesn't care about your open source. It doesn't care about logic, or slowly shooting itself in the foot, or alienating its base.

It cares about sales, profits, and the balance sheet. As long as it can make a profit, the people supporting and developing for its platform for free can go fly a kite. And it will continue to burn itself into the ground until it becomes a write-off. Because there is no founder worried about its livelihood to try to right the ship. There is merely an executive VP who needs to jigger the numbers to make it look like they got another win this year so he can make his bonus.

I call this Corporate Source.

Do not anthropomorphize the IBM.

I always come back to Bryan Cantrill telling me not to stick my arm into a lawnmower.

Amazing talk that's referenced in this thread. Good review of Sun's opensource history, and a hilarious amount of shade thrown at Oracle: https://www.youtube.com/watch?v=-zRN7XLCRhc

The specific clip starts at 33 minutes but I recommend watching the whole thing. Also Bryan is on HN!

It hates it when you do that.

and above all, never feed it after midnight

(1992) https://genius.com/Information-society-where-would-i-be-with...

This is an important point! Corporations don't make decisions - individual people do. A corporation is a piece of paper in a drawer.

Wait, you're telling me IBM isn't committed to their messaging of Peace Love Linux, 23 years later?

https://linuxgizmos.com/files/peace-love-linux.jpg

You're probably right, but for some of us (certainly me) Red Hat is like an old friend. If there's even a tiny chance we can stop them from running off of a cliff, we have to try. When we see the self-harm they are doing, it pains us.

Now, if there were widespread engineering dissatisfaction and turnover and they started losing some of the top talent and nerd-famous people that work there, I think there's a chance the people at the top would listen. Just a chance of course.

I think the much touted phrase of “free as in press not as in beer” will forever haunt the FOSS community. If you need a good example, take a look at the state of the “free press” and the depths they sink to for relevancy. The entire incentives of the market are at odds with ideals like those of FOSS. All you can do is bury your head in the sand about it, start a “company” that upholds it for a little while, sell or go public when the time is right, then watch it be everything you set it out not to be.

Or never sell, never go public, and fade into obscurity eventually with dignity. Despite how discipled and committed you are, you are mortal. A company is immortal. Even the most ideological of movements (e.g: religions) eventually evolve to resemble nothing of their original callings.

Fade into obscurity with dignity whenever you can.

I think the hope is that Red Hat would care about its own long term profitability. That is, the people running Red Hat will realize that they're sacrificing future revenue by prioritizing instant profits and this course is unsustainable. If they're interested in maximizing long term revenue, they should take the long term view.

That is a reasonable idea. Unfortunately, the situation seems to be that the execs making these decisions are taking a pump and dump approach to the brand and don't care if Red Hat is still in business by the end of the decade. They'd rather the certainty of a dollar today than the promise of ten dollars tomorrow.

> I think the hope is that Red Hat would care about its own long term profitability.

They are a public corporation. They only care about the short term sheet as that is all investors care about. As long as the stock market punishes long term thinking this is what you will get. Part of me thinks it is by design... that it is easier to make money in the market if you have a steady flow of companies going public then eventually going out of business.

> As long as the stock market punishes long term thinking this is what you will get.

I hate to have to ask this, but why is this not priced in? In a naive conception of the stock market, public knowledge that a company is screwing itself over is basically free money, and somebody should take that money and thereby push the stock price down. Why, again and again in tech, does that not happen? Is the slow decline simply too slow or what?

There are a few parts to this. The first is, the future is currently extremely uncertain and therefore short term moves make sense. The second is the same as every other industry, if the price of money (interest rates) is super low people accumulate debt leveraging the future for the present and thereby raising the need for immediate return. This second part creates a feedback loop of constantly needing more and more revenue to pay the lenders. The third part of this is that most megacorps become seriously dysfunctional over time. There are a ton of reasons for this, but it mostly comes down to growing bureaucratization. IBM is among the worst in this regard, and many people predicted exactly this future when IBM bought Red Hat.

Personally I see it as an inherent flaw in the corporate model and we should rethink the idea of permanent corporations.

There are Corporate and then there are IBM.

Some corporations have a working long term plan, even if those plans are are purely about making more money in the future. Can't say the same for IBM.

Everyone seems to forget Redhat is owned by IBM, and this is their core business model. IBM doesn't care that this guy is mad, they were never going to make money off him. If you're a bank (for example) running off Redhat, you aren't going to leave them because of this, and if you make software for banking companies (continuing the example), you'll pay up for the licensees to test your software on.

If you're open to moving to another distro (Rocky, Debian, Ubuntu, Alpine, etc), you were probably never to going to license Redhat anyways.

This is patently false. Redhat is making bank off of this guy. Jeff has written 100s of Ansible roles for free and made them widely available. I've used dozens of his roles as a contractor for F500 companies to help get projects off the ground quickly.

Now instead of giving him free binary compatible OSes to develop his roles against, he'll probably just drop support because there's too much hassle. The next time I go to install an OS, it'll be Debian, because most of the tools I'll use won't have packages for RHEL like OSes.

This is so dumb it hurts, Redhat won't recover from this. It'll cripple their package ecosystem over the next decade.

> It'll cripple their package ecosystem over the next decade.

but in a way that's really hard to measure and doesn't show up on a bean-counter's spreadsheet. So that means corporate will consider it "good business" and will just blame something else for the immeasurable failures a decade later.

>This is patently false. Redhat is making bank off of this guy. Jeff has written 100s of Ansible roles for free and made them widely available.

Firstly, creating value is not the same thing as creating revenue.

Second, people aren't licensing Redhat for access to these. People license Redhat so when they have a bug that affects them, it gets fixed right away with guarantees. Unless that guy is getting paid to do those fixes, I don't see how this affects Redhat customers.

You'd be surprised how many paying RHEL customers use community-contributed Ansible roles because there's no way for a company even as large as IBM to employ experts in all the various software RHEL users install to maintain integrations.

That's the long-term effect of starving the wider community: you lose having 'great' integrations, and you end up with poorly-written and barely-maintained integrations for anything that's not extremely mainstream, and so you start losing the customers who use anything outside the core set of products extensively.

I'll just copy paste what I said in another comment:

>I'm not saying this is the morally right, "best" in the long term sense, or otherwise socially optimal choice, I'm just saying it's the standard IBM choice.

Ah, missed that ;)

You make it sound as if it's without doubt that a) a significant number of RedHat users require Ansible and b) those users will stop using RedHat if this Ansible integration becomes unavailable.

I highly doubt both. I don't have numbers to back it up, but neither do I see any indication that it will matter to RedHats baseline.

>IBM doesn’t care that this guy is mad

No, but they should. Jeff Geerling is one of these guys: https://xkcd.com/2347/

IBM might not make money off you or me, but they absolutely make money off people like Jeff. In some regards, that’s kinda their whole business

IBM is a big company. They can pay one their 300,000 employees less than one average support license to replace all the work he does for them indirectly, and if he's opensourcing his software, it's not like he can stop them from using it even if he is mad.

I'm not saying this is the morally right, "best" in the long term sense, or otherwise socially optimal choice, I'm just saying it's the standard IBM choice.

> They can pay one their 300,000 employees less than one average support license to replace all the work he does for them indirectly

I bet they can’t. He’s hardly some code monkey churning out generic lines of code.

xd they definitely can't. maybe with a small team they could replace some.

Even if this causes RedHat to trend towards oblivion and 0 usage IBM will continue to make a shitload of money in the interim and IBM will trot out an army of MBAs with spreadsheets to explain why this change earned the company more money even if it caused the demise of RedHat.

You see, you are just being sentimental about trying to save something you're nostalgic for. This MBA army can clearly justify with mountains of spreadsheets why this was the Correct Thing To Do to maximize shareholder value and profit.

My first Linux installation was a copy of Red Hat 4 (not RHEL 4, RedHat 4), circa 1996-ish. I remember bugging my mother for weeks to drive me to CompUSA and buy a copy. I am deeply nostalgic.

> If you're a bank (for example) running off Redhat, you aren't going to leave them because of this

Not sure if it ever happened, but when I was at BofA there was definitely interest from up high on going to Oracle Linux

Out of the frying pan and into....

It's ironic my org switched from AIX + websphere to Red Hat + Jboss I believe due to licensing.

The IBM owners / shareholders probably aren't enthusiastic. Unless short like Chanos https://app.hedgeye.com/insights/94584-jim-chanos-on-hedgeye...

Rocky Linux listing the following (funny) options:

- Buy a farm in Nebraska or Montana. Raise cattle Stack suggests goats instead, as they may be easier (plus milk, cheese, gyro's, and they will eat poison ivy!) - Utilize RH subscriptions to access CDN sources and create local source mirrors for every package which we can then import from Q for legal: we need to review the licensing terms - Infiltrate RH

https://etherpad.opendev.org/p/r.24fab14385c0aa2db6fa7340a8b...

Author of that pad here: For the record, someone changed their color to something similar to mine and added the 'Infiltrate RH' line which has been since removed.

But thank you for appreciating my humor lol

I don't think Red Hat is trying to kill clones, but I believe they're trying to kill clones that are "1:1 compatible with RHEL".

They're trying to force a shift in what clones are supposed to be, and that started with CentOS Stream. Locking the sources to RHEL-proper is just the next step.

This may signal their death as a widely-recognizable Linux distribution vendor, but I can't really say I blame them.

If CentOS Stream isn't adequate for production — and I see it as being at least as adequate as Debian for those that also upgraded from one RHEL major to the next soon after its release — then the future of clones may lie in becoming downstreams of CentOS just like RHEL is.

I do see the value in a CentOS Stream derivative that's more conservative, but it doesn't need to be 1:1 bug compatible with RHEL.

Honestly, I don't see the point of 1:1 clones unless you're running some closed-source commercial software. And such software tends to not be supported by its vendor even on 1:1 compatible RHEL clones.

I like the Red Hat flavor of Linux, and I like the fact that I can have a machine with an OS installed in 2014 continuing to get security fixes without having to upgrade major versions. But I don't care if that's 1:1 compatible with RHEL (and it doesn't really need to be supported for 10 years).

Well, for those people who like upgrading, CentOS Stream still doesn't support in place upgrades.

Wait, dnf system-upgrade won’t work?

Granted, CentOS Stream has only seen two major releases in its current form, but still.

AFAIK yes, neither on RHEL. There is no official way to do in place upgrades on CentOS stream.

Found something on Reddit to that extent.

Since AlmaLinux was working on a tool to allow just that — and RH Engineering told that they themselves are not interested in in-place upgrades but welcome the community to do so — crippling the said community is a double dick move.

Now, where’s that escaped lion roaming IBM office corridors when it’s needed…

>>clones unless you're running some closed-source commercial software. And such software tends to not be supported by its vendor even on 1:1 compatible RHEL clones.

this is false, there are TONS of commercial software that runs certified by the vendor to run on RHEL supported by the vendor on RHEL, and only supported by the vendor if you run RHEL, that is one of the things that made RedHat into a billion dollar business.

ERP systems, commerical databases (that many here probally have never heard of), and tons of other enterprise software.

Many of these vendors have been adding Ubuntu support since the CentOS changes.

That’s exactly what I said: even if you’re running a 1:1 clone, they won’t support you unless it’s RHEL itself.

what I said

"Many of these vendors have been adding Ubuntu support since the CentOS changes."

> I see it as being at least as adequate as Debian for those that also upgraded from one RHEL major to the next soon after its release

What's wrong with Debian? I assume this is from a stability standpoint?

Nothing wrong with Debian, but RHEL has a better support cycle and more engineering resources dedicated to that support.

Debian 12 will be supported for 5 years, and so will each CentOS Stream release (it ends with the release of the last RHEL point update).

RHEL is supported for 10 years (the last point update is supported for 5 additional years).

I think a lot of companies are using RHEL and Alma/Rocky side by side. Because they are compatible, because people have an easy time to switch between them. I don't think that those people will move their license free servers to RHEL, but they will probably switch to Debian/Ubuntu.

So in the end Red Hat will have a smaller market share and less support from the community. Currently most software provides a package at least for RHEL and Debian/Ubuntu. That may change in the future, if the amount of RHEL compatible systems drop significantly.

Noone is switching from RHEL to Ubuntu or Debian. If you're on RHEL it's because you need their ridiculous long support cycle which noone else comes close for.

It's used by people building medical hardware and telescopes, not for website deployments.

I meant their license free Alma/Rocky Servers.

Sometimes it's because you're operating a platform for a government entity and they have requirements like 'all OSes are under support contracts'.

It's harder to get reputable support for debian in the south Pacific. RH is damned near the only decent option.

Or compliance certifications. Whomever has the one you need, that's the distro you're going with. And any customer who needs that compliance usually has a boat load of cash.

Microsoft really missed an opportunity for a boatload of cash by not putting out a distro for enterprise and government use. I guess they're making up for it now on Azure.

If you're running some expensive software, the few hundred bucks for a RHEL license doesn't matter at all. Some database licenses easily reach a six figure price per server. Custom built software often runs only on a handful of servers and can then easily reach seven figures per server.

Ubuntu offers 10 years which is the same. RH is more entrenched, especially in the US. On the technical side, aside from selinux v/s apparmor, there aren't too many differences between Ubuntu and RHEL. RH has a few more custom offerings on the server side (freeipa, openstack, rhev, gluster, etc.), but those are probably not relevant for most users.

Ubuntu has 5 years of “standard support”, with EOL after 10 years. RHEL has 10 years of “production”, with another 3-5 years of “extended support”.

That's just corporate speak. They are all the same. If you use RHEL after 5 years or after the .5 release, it becomes plainly obvious that it's begging you to pull the plug. It's only half assed security fixes after a few years, no matter where you are. If a customer asks for 13 years, I'm sure Canonical will oblige.

Edit: I'm talking about Ubuntu Pro that is 10 years.

It's that "corporate speak" that wins paid contracts for software that needs to run for a decade or more.

RHEL competitor in the market is Windows, not half supported hobby distros.

Ubuntu PRO is a subscription with 10 year support for Ubuntu main and Universe repos, so more than 10 thousand packages. It's also much cheaper, and comes with more features like kernel live patching.

Ubuntu is not a “hobby distro”. I do not even like Canonical but let’s be real. First, Ubuntu is used as the basis for just a giant fraction of enterprise software in the cloud and in embedded. Second, companies pay Canonical big money for even their managed Kafka offerings and such. Canonical is a legitimate business alternative. They are not peddling 10 year support for the hobbyists.

Ah, that makes sense. As somebody outside the RHEL ecosystem, my main experience is needing to argue that maintaining compatibility with whatever ancient version of gcc ships with the oldest “production” version of RHEL isn’t worth it. (E.g. that it is okay to require C++14 support)

I can't find numbers, but an awful lot of customers are still on RHEL 7 which was released in 2014. It's old but it's way more up to date than Ubuntu 14.04.

I myself only switched (to CentOS Stream 9) last February for my personal servers.

Canonical started that 10 year thing only recently (maybe around 18.04). Also, you need to be on the pro subscription to get updates beyond 5 years. So a more accurate comparison would be 18.04+pro and RHEL 8, accounting for the fact that RHEL 8 is a year newer. Or 22.04 and RHEL 9.

Still on RHEL 7? We upgraded to RHEL 7 from RHEL 6x three months ago :-)

FreeIPA is a compelling differentiator.

I've just started testing the port that recently landed in NixOS.

FreeIPA is nice, if over engineered and fragile. There is nothing preventing Canonical from packaging it, they just don't see enough demand probably, and for the most part, they care about ubuntu clients joining an AD domain, which I think Canonical supports.

SuSE offers 10+ years support windows, is also RPM based, and has been the RHEL of Europe really for decades.

But support from the community almost doesnt exist.

Indeed, SuSE is a much more likely alternative than others that are being thrown around.

Fwiw, we moved (are moving) on from RHEL and derivatives to Debian/ Ubuntu servers at $DAYJOB. We were planning on deploying Centos 8 before the Stream debacle came along.

We did... Enterprise ERP systems...

Incorrect, for the longest time the US government only knew how to support linux via RHEL, that is rapidly changing. So if you sold software or services to the US government that was running on linux, your only viable choice was RH. That has started to change, Ubuntu pro is certified for use in the USG and has STIGs, which are hardening requirements for Ubuntu.

Many places would develop on centos or scientific Linux and then when it went into production use RHEL, since they were effectively 1:1 ports. So yea, I imagine there are quite a few orgs who might switch.

Given my experience, RHEL is a favorite among enterprise environments. As such IBM will continue to pursue a strategy that forces people into that business funnel. They do not care who falls out of it, the ecosystem will only become more closed.

I think we need to be more realistic about where things are headed. SUSE? Market cap of 2.2 Billion, IBM paid 34 for RH. Ubuntu? Still not profitable. As we continue to see the tech sector cool and become more mature belts will tighten and things will consolidate.

Can you elaborate why a market cap of 2.2 billion makes SUSE not a viable alternative? If anything, this makes me feel like they're undervalued, especially now that they're the only company shipping an enterprise linux with an open source and privacy respecting (looking at you Canonical) base distro.

What I meant is what would prevent IBM or other large corporation from acquiring them as well?

They already have been, multiple times, nowadays they belong to Micro Focus.

In July 2018, Micro Focus International, SUSE's parent company since 2014, announced its plan to sell the business unit to a subsidiary of EQT Partners in the first quarter of calendar year 2019.[7][8] This acquisition was completed on 15 March 2019, making SUSE a standalone business. Under new ownership, their legal name is SUSE Software Solutions Germany GmbH.

I stand corrected, it is still owned by them in 75% though.

SuSE is owned by MicroFocus.

SUSE was sold to EQT in 2019 and went public in 2021.

I stand corrected, it is still owned by them in 75% though, and apparently they are an investment fund.

Sometimes I feel that a lot of tech companies are virtually behaving as hedge funds

Do you have a source for that? I was under the impression SUSE no longer had any ties to Micro Focus

Wikipedia?

Are you confusing EQT and Micro Focus?

    SuSE is owned by MicroFocus.
    ...
    I stand corrected, it is still owned by them in 75% though

As per the Wikipedia article, EQT still owns 75% of SUSE, not Micro Focus. Micro Focus is no longer involved with SUSE

> Can you elaborate why a market cap of 2.2 billion makes SUSE not a viable alternative?

I'd always assumed the MicroFocus acquisition of SuSE had something to do with OpenStack, and when OpenStack stopped being The Big New Thing, that impacted SuSE.

Basically:

* I was on the HP OpenStack team

* HP burned through over a billion on that, and sold the product off to SuSE

* Somewhere in the mix there, MicroFocus was divested from HP. Even working there, it was hard to follow, because HP also split into HP and HPE

* As various vendors lost interest in OpenStack (Cisco, EMC, VMWare, etc), SuSE replaced their CEO who was behind the acquisition in the first place

I talked to some SuSE folks about OpenStack at one of the VMWare conferences, and they seemed confident in it's future, and then just weeks later I read that they'd discontinued the product in it's current state

The entire thing is incredibly confusing because HPE also had two versions of OpenStack. "Helion Open Stack" and "Carrier Grade OpenStack" and they were two different teams.

I didn't think they really had to pay for the openstack assets, I didn't think they did anything with those. IIRC they had their own managed openstack contracts with their SLES customers before the acquisition, and were working on getting kube/paas offerings on top of that which they then scrapped for Rancher.

So I know they burned a bunch of money too, but I don't think it was on the HPC or other openstack assets (HPC was the public cloud openstack offering that I think got shutdown sometime around the HPE split)

I bet Microsoft will eventually buy Ubuntu, given their relationship on WSL and key distribution for Azure.

I've suspected this for a while as well. At one point GOOG would have been a good bet, but having turned down many potentials over the years they clearly don't have an interest. I could see MS buying Canonical (or Ubuntu), although there has to be enormous pressure against that from the Windows people.

Why should be pressure be from Windows people?

They just announced at BUILD 2023 the graduation of CBL-Mariner Linux distribution into "Azure Linux", and guess what else has same support level, Ubuntu.

https://build.microsoft.com/en-US/sessions/4c586f76-ad6e-444...

Naturally, they have plenty of other distributions being supported.

If anything, most likely it is the Azure focus that is partially responsible for the chaos of Windows GUI frameworks, as the teams have minimal resources to keep going.

But why? What would they gain? You think Microsoft is interested in Linux enterprise support?

Microsoft is very interested in any enterprise support it can monetize. Ignoring WSL (you shouldn't), you'd be surprised how much Linux runs in Azure. Buying Ubuntu lets M$ say "sure you can run Linux in our cloud. We have one right here, one we support, for a modest fee".

Yeah, but we all know Ubuntu is used because its beginner-friendly and free. Once it becomes paid, poof everyone switches to Debian.

Ubuntu is massive on cloud platforms. I'd bet bigger than even Debian. And IIRC even AKS runs on Ubuntu. At least last time I used it.

Because for many people, Ubuntu is the linux distro.

Why? Because it's free, and very simple to use, but if it becomes paid, no matter how good it is, devs will switch to something else, and the market will follow them, gradually.

That's an extremely reductionist/simplistic view of why people pick, use and/or discard an operating system. And, FWIW, Ubuntu isn't my primary cloud Linux.

Unless you can demonstrate to Microsoft that enterprise desktop Linux support ROI can ever rivals Window’s or their cloud bunsiness, they won’t be interested.

In fact, I’m pretty sure Microsoft doesn’t know what to do with the Windows business at this point. They really would love to dump it, but they can’t. Office is more profitable than Windows. Azure is more profitable that Office and Windows. If trends keep, Xbox will over take Windows. There is no growth potential anywhere in the Windows business. They tried hard with the Surface line to imitate what Apple has with OSX, but they failed. Surface didn’t become the de facto Windows hardware like the Mac is for OSX. It’s just a side hussle to see where it could lead in the future for them.

Microsoft already announced their “Azure Linux” distribution which is like Amazon is a fedora based Linux flavor that just lets them control the versioning of all user space rpm packages for a server/docker offering in Azure.

they have no interest in making this a full “Enterprise Linux” offering. as long as they can offer servers an docker images that are rpm compatible, they are good for now.

WSL is entirely kernel layer. So whatever your distro is, they are good with it.

Most of the threads here - where people describe the corporately driven business wants/desires - seem to recognize though that this is killing the holden goose.

Enterprises want RHEL because the staff know & use RHEL and the enterprise can get a support package for RHEL. If Red Hat no longer is in the public eye, if it's only a paid for Linux, the demand is going to dry up, vanish.

There's only one way to retain mindshare & market share & that's to keep RHEL in front of people. This current course of action almost guarantees that RHEL fades into irrelevance.

AIX would like to have a word...

Exactly. They will likely reduce costs and milk it as long as possible:

https://www.theregister.com/2023/01/12/ibm_aix_developer_job...

If you are using RH, or SUSE, it is because pointy hairs forced you to use RH or SUSE. So you are kind of trapped in that funnel.

SUSE is tenacious AF, you have to give 'em that. This could be an opportunity for them to grab some market share. I'd love to see it. I don't wish ill on Red Hat, either, but I'd love to see SUSE grow and be a stronger player here. It'd be good for the larger ecosystem.

About 15 years ago the (German) company I worked at had a lot of their customers from different industries run on SLES. But over time many customers just weren't willing to sign up for OS support contracts anymore (and also switched from Oracle to MySQL). SLES/openSUSE never managed to get the same support from third parties and communities as CentOS/Fedora. With RHEL 6 we migrated all customers from SLES to CentOS/RHEL. This allowed to have a homogenous environment with and without enterprise support contracts.

We'll have to see where this is going. Certain enterprise software (like Oracle) requires the use of certified operating systems or you lose support. Oracle Linux looks like the more natural choice.

Red Hat, you made the internet's nicest man, Jeff Geerling, upset and you should be ashamed of yourselves.

"Many people believe that the spirit of the GNU Project is that you should not charge money for distributing copies of software, or that you should charge as little as possible—just enough to cover the cost. This is a misunderstanding.

Actually, we encourage people who redistribute free software to charge as much as they wish or can. If a license does not permit users to make copies and sell them, it is a nonfree license. If this seems surprising to you, please read on."

https://www.gnu.org/philosophy/selling.html

I believe that it would stand to reason that if IBM/RH is not permitting redistribution of sources made available in their customer portal, this would constitute a violation of the GPL for any GPL package.

I would assume that IBM/RH will then need to take the Apple approach and quickly start working to replace GCC with LLVM and next rid itself of all GPL software and stick with MIT/BSD/OSC...

They do not restrict redistribution of what you already have access to, but they don't give access to future updates if you do redistribute. The GPL doesn't require them to give access to future updates either.

That is an excellent point I hadn’t thought of.

As an ex Red Hatter (pre IBM) this feels like top-down IBM influence, and against the always open spirit of the old Red Hat.

I'm also an ex-er but was there post acqui. It's definitely IBM influence, but it's long-time Red Hat people who are actually making these decisions. I don't think these are people that actually believed in Red Hat in the first place. Or if they did, they're willing to IBM-ify in order to keep their jobs (or ascend the ladder)

I have to agree. All IBM is asking for is to hit certain numbers. Now, those numbers might be short sighted and unrealistic, but in my experience the operational decisions were still coming from inside Red Hat, at least through the middle of last year when I left. It doesn't matter whether IBM or Wall Street is demanding double-digit growth numbers every quarter, the end result is the same. Despite a really strong push by a lot of people inside the company to put culture first, in the end, a demand for unsustainable growth is going to wreck any culture, no matter how strong.

It's exactly this. When I worked at IBM some years ago via another acquisition, this is the same pattern I saw. Senior leadership -- people who had been at the prior company for nearly two decades -- were the ones gladly steering it aground in order to do what IBM wanted.

You can see this with Red Hat - the Chairman and the CEO are both long-time RHers. Were they just the opportunists?

All the talk about running RH as an independent subsidiary was either smoke and mirrors or the IBM management style has infected the leadership at Red Hat.

> or the IBM management style has infected the leadership at Red Hat.

Tbh, if it hadn't, they would have been outed long ago.

I'm a former Red Hatter. I left just after the IBM acquisition. We all saw stupid stuff like this coming. It's a sad loss, but the Debian (and Debian-based) world is lovely.

As a long time Debian user for 20+ years I have to say I really prefer dnf/yum over apt/dpkg now. It's significantly faster and has much less version deadlock snafus from my experience in the last 5 years. I still use and support servers from both sides and my Debian servers are often a chore to work with by comparison.

Corporate brain fucking up yet another formerly admired organization.

RIP Redhat. And fuck you, IBM.

Everybody treats the sale of a company like you’ve won a game of Life. I went through a few where I obviously did not become a millionaire, and with a lot of time and unpacking it with coworkers and people with similar experiences, it’s more like an Irish Wake.

Sure there’s booze and laughter, but something unique has been lost to the world, and that’s why we are all here.

Quick tip for any AWS users (or users of any cloud provider, really): if this affects you, let your account manager know, along with your thoughts on how you anticipate responding. That feedback is invaluable for service teams who are trying to figure out what OS support to prioritize for the tools they might be building.

Recent and related:

Impact of RHEL Changes to AlmaLinux - https://news.ycombinator.com/item?id=36436375 - June 2023 (43 comments)

Red Hat cutting back RHEL source availability - https://news.ycombinator.com/item?id=36420259 - June 2023 (299 comments)

Red Hat Now Limiting RHEL Sources to CentOS Stream - https://news.ycombinator.com/item?id=36419586 - June 2023 (27 comments)

"Any headline that ends in a question mark can be answered by the word no."

i worked in many different companies (including multi-billion revenue per year), and all of them used centos in order to have "proper enterprise linux" but for free. The only cases when real redhat was used is for something that actually required it by licensing/support terms - like Oracle, or alternatively contractual obligations for SLA.

Pretty sure that IBM/Redhat know this as well and "third party open source packages compatability" not exactly something that they care about. Availability of sources only takes business from them.

I personally somewhat happy about it. I moved to Debian 20 years ago and it "uncomfortable" to live in RH derived world. Even though Redhat 4 (or 4.1) was first ever Linux distribution that I installed back in 1997

PS. Also when RedHat "partnered" with CentOS back in 2014, it was somewhat obvious what is the direction. If there is anything that it surprising, then it will be that it took so much time.

> i worked in many different companies, and all of them used centos in order to have "proper enterprise linux" but for free. The only cases when real redhat was used is for something that actually required it by licensing/support terms - like Oracle, or alternatively contractual obligations for SLA.

I worked on a product that allowed both RHEL and CentOS, and then moved to a different one that mandated RHEL with a supported contract. The latter was far less stressful to deal with, because any time there was an OS issue, we could pretty easily tell them, "Go to Red Hat." The shops that were on CentOS on the other product tended to be small businesses (and cheapskates), and every time an OS issue arose, it was a screaming match trying to convince them that we're not going to fix their OS for them.

i just edited my post when you replied. companies where i worked had multi-billion revenues. it's not exactly small business. they just don't want to pay money for proper redhat. they had money to do so, but savings are savings.

Red Hat also conducts software audits.

That is a huge reason to avoid them, regardless of the savings.

in some cases, it's just cost of doing business

RHEL market share won't last forever, so they have decided to milk the Red Hat cow as fast as they can.

Long term, people will switch to containers and cloud apps (Kubernetes, AWS). In this new environment, operating systems are commoditized and RHEL's moat will slowly shrink. Instead, developers will focus on alpine, debian or similar.

Red Hat has significant involvement in container platforms right now.

Those are different Red Hat products and belong to different layers of the cloud stack.

Previously third party developers targeted RHEL directly, enhancing the RHEL ecosystem. Red Hat could play long term by (indirectly) giving away the base product to the community, and profit from the increase in support subscriptions.

New projects don't target RHEL as often as their primary platform, they prefer containers now. There is less ecosystem enhancement and it's more profitable to milk existing RHEL users. Centos Stream makes it possible to smooth the transition to the new paradigm.

Openshift, Coreos, etc have different purposes and follow different market dynamics. Mainstream developers don't target them directly.

My company is steadily moving from being a heavy RH shop (RHEL and OpenShift) to Amazon (Amazon Linux 2 and EKS) as we migrate from our colos to AWS. RH has been a good vendor and I have no complaints, but it makes little sense in the long run. I am sure they are feeling it.

Seems like we're in a period of ecosystems shooting themselves in the foot, be it 'social' media, software, banking or politics.

I suppose, in a way, we have gotten accustomed to a relatively long period of stability. As for what RHEL thinks they are doing with this: every time a (IMO: nasty) enterprise vendor starts packaging their software in containers (IMO: a good thing!), distributions become less relevant by the day, and so does their view on software distribution.

We’re in a period of unsustainably subsidized free environments being culled.

People trying to monetize free things in order to continue providing free things seems to not be working all that well.

The thing you do to make money and the free thing you offer probably just need to be separate for long term stability.

The Linux kernel model of lots of institutions supporting an actually free project is probably the only real long term successful free model.

I suppose the linux kernel model has the benefit of having a mutual dependency so you can't really easily long-term extract value without adding value. (i.e. forking the kernel means you'll be spending a lot of time keeping up, so doing to work to upstream your stuff has a real benefit)

Ironically, Red Hat is one of the biggest supporters of Linux Kernel and now needs to actually fund that support and dev time.

Business models break, when money is no longer almost free. Interest rates near zero created a historically rather unique environment. That’s now apparently ending.

> Seems like we're in a period of ecosystems shooting themselves in the foot, be it 'social' media, software, banking or politics.

Examples of companies besides Red Hat that have also been burning huge amounts of good will lately indeed come to mind easily: Activision Blizzard, Reddit, Netflix, …

Reminds me of the wise words of Steve Jobs, the sales and marketing people are running things now. The product Genius has been rotted out, by people who have no conception of good product, no feeling in their heart about wanting to help customers.

https://www.youtube.com/watch?v=P4VBqTViEx4

> When Red Hat decided to turn the community CentOS distribution into a leading-edge distro instead of basically "Red Hat Enterprise Linux, but free", users like me were justifiably angered.

I miss CentOS too. But how is that justifiable anger? Are you angry at the myriad of companies using open source code to build a paid product, and not giving it for free? I absolutely think that it backfired and they shouldn't have done it, but I also understand they had no obligation to maintain a 1:1 free alternative of their main product. That's just an unrealistic expectation imo.

It's not comparable, as the source code is mostly not theirs.

As long as they comply with the licenses, which they do, that's largely irrelevant I'm afraid.

I'm a little confused how this new move still complies with many licences.

As a paying customer you receive access the sources, as you're supposed to - but under contractual terms that you can't share them.

Take the GPL as the most famous example. Providing me GPL'd sources that I'm not allowed to redistribute under the same terms definitely feels like it's against the spirit of the agreement. I'm too warm-blooded to argue whether it's under the letter.

It's been said elsewhere, you can legally redistribute, but then they will end your subscription, as those are the terms you agree to.

So the restriction isn't on exercising your rights on what you've received, which would be under the terms of the licence - it's on their willingness to provide you with anything else, which would be under the terms of their contract.

I think I'll stick with my position that it's not within the spirit of the licence. (But I'd agree that this change has no impact on this.)

Not just the spirit, the GPLv2 terms include:

"Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein."[1]

So if they forbid sharing, they can't distribute Linux themselves.

[1] https://www.gnu.org/licenses/old-licenses/gpl-2.0.html

It's legally irrelevant, but if you want an explanation for "justifiable anger" it's extremely relevant.

These days isn't OpenShift their main product?

Changing the support terms from 10 years to 1 year mid-stream lost them a lot of trust. It's not just what they did, it's how.

To that, I couldn't agree more.

Redhat has some packaging scripts that detect rpath and refuse to install binaries that set it. This has caused me a good couple of days of grief and lead directly to shipping a worse product. Specifically, clang -fopenmp a.cpp -o a.out used to give you a binary that would run with ./a.out, but now it'll give you something that complains that it can't find the openmp runtime library. Unless you happen to have llvm installed globally, like redhat presumably does.

I now cheerfully try to talk people into dropping IBM Linux support whenever it comes up in discussion. Haven't succeeded yet but will keep trying. I want it out of the requirements graph. I liked rpath and will not forget the unjustified line of reasoning behind disabling it.

This is a pretty good opportunity for Canonical to pick up the slack. Aside from the snap related self-sabotage, Canonical was on a pretty good trajectory.

Don't forget baking Amazon ads into their default desktop environment. Unity, was it? I forget which one.

Given their team up with Microsoft on Azure I don't see a redemption arc anytime soon. What is left, SUSE I suppose for an enterprise style LTS distro?

I have been using debian and ubuntu for almost two decades without using Redhat, things seems OK, will this change by Redhat push more users to debian|ubuntu side?

Canonical probably will do something similar in the future, but at least we have Debian.

I mean they might've already done it, is Ubuntu Pros source code open?

Didn't expect the RaspberryPi YouTube guy…

NOBODY expects the RaspberryPi YouTube guy! His chief weapon is surprise. Surprise and absurdly impractical Kubernetes clusters. His TWO weapons.. etc.

Geerling is one of those guys who speaks with authority, mainly by not speaking unless he's sure he's an authority on the matter. I've solved many problems by just following his youtube antics and blog posts. He's also kind of an Hn rockstar and lurks here on occassion.

Aww, thanks.

I think for me, this whole thing hurts the most because I know SO many people in the Red Hat ecosystem. From Fedora, to Ansible, to RHEL, to Kubernetes/Openshift people... I've worked closely with so many.

And there are still many who work there, and I know the words I wrote will sting for some of them. I don't hold any of this against them.

It's just... painful to see the Red Hat that I remember from a decade ago, slowly becoming, what? IBM-ified? There's still a lot of good in that company, but it is being blotted out over time.

Funny, I mostly knew him as the Ansible guy. He wouldn't be notorious without RH and look at him now.

Notorious? I didn't know he was notorious at all..?

In here, kind of.

I thought you meant Explaining Computers[1] and had to go back and check.

https://www.youtube.com/channel/UCbiGcwDWZjz05njNPrJU7jA

"yes"

That's today's episode of EZ answers to EZ questions.

What a great episode, see you on the next!

As I'm reading the comments I'm a bit curious what services people are running, or dependent on CentOS for, that FreeBSD or Debian can not provide? In the past I could see buying into an eco-system of tooling that helped you manage fleets of servers but with Ansible, Docker, etc wouldn't the tooling be agnostic to the OS?

What exactly is it that you are running in your infrastructure that doesn't have a stable alternative?

- DNS - DHCP - Webservers - Database servers - Firewalls - IDS - VPN - Auth services

You can PXE boot, use tools like foreman, upload images to VPS provider(s). At least speaking for FreeBSD you have regular release cycles, security advisories, etc.

I'm probably old but this scenario has played out plenty of times and the comments are the same that were on mailing lists when RedHat Linux was moved to Fedora. I remember when the CentOS maintainer(s?) disappeared and delays between CentOS 5/6 and RHEL6. People talked about how their business were so dependent on this OS how could a maintainer just abandon it..etc.

Now resources and efforts are split between Alma and Rocky linux at what point do you evaluate what you "really" need from your systems and if the constant wait-and-see, buy-out, merge, fork process just becomes noise.

Has it never come up in any risk assessment for large companies to have a contingency if the OS is no longer available or the licensing changes?

> Docker

FreeBSD is lacking support of Docker - they prefer to think they can ingest system management knowledge into Frontend Developers and suggest to use Jails (with ZFS/Nullfs/VNET/other horrible words for Joe The Svelte Dev )

> upload images to VPS

Some still have things running on premises

> dependent on CentOS for, that FreeBSD or Debian can not provide

Good luck running SAP products on this

> FreeBSD you have regular release cycles, security advisories

FreeBSD lacks the idea of LTS. At most you (to my knowledge) you may have updates to a) the very minimal set of things aka base system b) it ends in ~ 5 years, not even 7 or 10 c) someone will need to the the job to keep security support for ports tree for 5 years to ensure versions are kept the same. Or in other words - requires more human power on your side comparing to offloading that to the vendor.

For some FreeBSD guys example may be needed.

Imagine you installed latest Ubuntu LTS (22.04) which has Nginx (as part of the whole distribution provided, not "base"/"ports" separated) and it's version is 1.18 and for next 5+ years it will keep the same version. No surprises. You even run auto-upgrade everynight and reboot server ~ once a month for kernel updates.

Then repeat the same on FreeBSD - the ports tree will get updated with newer Nginx ( I've checked https://ports.freebsd.org/cgi/ports.cgi?query=nginx&stype=al... here, it's 1.24.0 and no older versions available) and when 1.25.1 will be become new mainline (probably in 1.26?) your configuration will stop working as they have deprecated "ssl on;" construct.

Or, other sample - let's say I'm developing Nginx module and selling it in binary form. I realize that my auditory is Centos/Ubuntu users. I know in advance in that case, which compatibility I need to support in terms of ABI, features, support articles and training. It helps me [as vendor of 3rd party software] with planning and keeping lower costs.

From reading your response I think there are more questions than answers. I'll quickly go through the first ones and get to the main items.

1. Docker does run on Debian[1][2] which is why I referenced "Freebsd or Debian." For reference I've run and used jails for a number of years with development, staging, and production environments. I don't know why frontend developers would be doing system administration, setting security controls, or that using Docker would count as "system management knowledge."

2. I spoke to the on premise part with foreman[2] and pxe booting. The VPS uploading was in comparison to the default images already available for linux with cloud providers verse FreeBSD.

Generally speaking, on your FreeBSD example and most of your points, it reads to me as regular system administration. You can manage your ports and packages just like having .rpm or .dep repos with poudriere[4]. You can lock specific versions of software from automatic updates or not. I'm not really sure why 5 years isn't enough time for a base OS to be supported since you can build your ports from source at anytime. Yes you are required to check the UPDATING file for any breaking changes however if nginx or similar application is critical to your operation and you are speaking to using Docker and other build tools wouldn't you be testing software upgrades anyway? As a vendor wouldn't you have nightly builds to download the latest nginx version and test it? It would seem a bit naive to just "trust" that an auto update would never cause issues because its in LTS.

The bigger item to address, if you are purchasing and using SAP products as part of your business or are a vendor producing/selling a module I don't think the CentOS complaint holds water. The risk you have accepted is that you have no control over how the support, development or updates progress but you want no surprises because it affects your ability to get revenue. That would appear, to me, to be a cost you should pass on to your customer, maybe they will move over to Ubuntu and make things a bit easier for you support and training wise. Most people choose CentOS because it was free and not RHEL thats the risk taken and sometimes risks come with new costs. If your module(in this example) makes them enough revenue, similar to anyone running SAP products, they should have the budget to pay for it.

[1] https://docs.docker.com/desktop/install/linux-install/ [2] https://docs.docker.com/engine/install/ [3] https://www.theforeman.org/ [4] https://github.com/freebsd/poudriere

> Docker would count as "system management knowledge."

Jails count as it. Not even going deeper into interesting questions - forcing team to ditch Docker and start using Jails.

> You can manage your ports and packages just like having .rpm or .dep repos with poudriere[4]

Interesting, I'm giving you examples of working _less_ and you saying - but you can work more!

And no, I/one cannot - who will backport patches for security issues? who even will be reliably track what kind of patches needed, what applied, how to test it and so on. Sounds like a extra spending on hiring some nerds focused on security stuff in this, instead of delegating to vendors.

> Yes you are required to check the UPDATING file for any breaking changes

Thanks, no, again. Sounds like LET'S WORK MORE again. Reading on changes will happen on next LTS version deployment and testing, not as day-to-day activity. Which happens with 2 year cadence and can be planned/prepared for migration.

1. Yeah the jails item was addressed above with Debian being able to run docker so don't go with FreeBSD for that. Again, you don't have to run the same OS for everything. Its not like homogenous environments haven't existed before, organizations run Windows, BSD and Linux together.

2. Plenty of organization manage internal repositories for packages they regularly use or for systems that need pinned versions of software. I haven't worked at any organization that hasn't had this. This was the entire benefit of having integrated system commands like yum, apt, fetch, pkg, etc and using .rpm,.dep, etc and standard package managers / formats. Not all companies release code to upstream repositories how do you think they systematically deploy their code from build tools like bamboo or jenkins?

3. Not even sure what you are talking about here. You don't have to run package updates everyday, you don't have to upgrade your FreeBSD OS everyday, now a 2 year release cycle is ok? but a (5) year supported FreeBSD release cycle is too short? Even if you outsource this to vendors you don't read the release notes, you just upgrade, you don't have to make any backups of your data or software?

At this point I'm not even sure what you do. You write software, it compiles, and it just magically shows up on the Internet?

My point still stands though, there isn't any specific item tying most of the people in this thread to CentOS that they couldn't pay RH or move to Debian/Ubuntu/FreeBSD. CentOS was never free from any of the items you noted above.

Let's address the last one, to make things bit more transparent

> My point still stands though, there isn't any specific item tying most of the people in this thread to CentOS that they couldn't pay RH or move to Debian/Ubuntu/FreeBSD. CentOS was never free from any of the items you noted above.

Hard to say on most people - I have gut feeling most of the population of HN is closer to development than operations and care on Docker/containers as platform, not the OS itself.

Agree on RH, Ubuntu. Don't agree on FreeBSD (due to reasons in previous replies) and largely don't agree on Debian. Debian has a short lifecycle and till recently required extra efforts on firmware blobs. I've superseded Debian to Ubuntu releases after Debian 8 or 9 for myself.

Point is on FreeBSD - it's total outnumbered hero in this list.

> now a 2 year release cycle is ok? but a (5) year supported FreeBSD release cycle is too short?

Right. 2 year (for Ubuntu, not for RHEL family) cadence doesn't automatically deprecate older releases - those still supported. So if you are running on 18.04 you can migrate to 22.04, no need to spend time migrating to 20.04.

For RHEL based, cadence is lower/more time between releases - RHEL 7 on 2014 and RHEL 8 on 2019, RHEL 9 on 2022. Real case, I observe right now, is future migration of Centos 7 systems to something RHEL9 based - Alma or Rocky - skipping RHEL8 base altogether.

> Not all companies release code to upstream repositories how do you think they systematically deploy their code from build tools like bamboo or jenkins?

That's more or less valid point for software out of your distribution's collection - like say Nginx modules which are not prepackaged or much more rarely - it's own software. Much more often home-made software is running in some sort of containers and packaged in Docker images, skipping "distribution" packaging phase.

> 3. Not even sure what you are talking about here. You don't have to run package updates everyday, you don't have to upgrade your FreeBSD OS everyday

I tend to run unattended-upgrades on Ubuntu everyday (by cron) and monitoring checking for security updates pending for more than 2 days, including "need-to-reboot" sign. I'd expect something similar guys in FreeBSD land do (probably `pkg audit -q` or like that)

Can someone explain how it is allowed to prohibit republishing gpl sources? I tho7ght the gpl expressly forbade this behavior?

Nothing unexpected after IBM takeover sadly.

Pick Debian for your next install.

As it stands I really like Fedora and I'm inclined to continue to give it a shot. I hope I don't have to eat my words on that. I believe that the Fedora Project is somewhat independent of RedHat, although they have a sponsorship arrangement.

Its relatively independent, but I don't see how fedora can be a RHEL replacement.

It can if you like upgrading your entire operating system every 6 months.

It is completely mystifying to me that anyone would use a "commercial" version of linux at all. I can't image commercial linux support is better than google and s.*overflow. I guess the answer is, as always, lawyers.

> I can't image commercial linux support is better than google

You get a real expert engineer answering your questions and even fixing package source verses an outdated answer on SO.

> I can't image commercial linux support is better than google and s.*overflow. I guess the answer is, as always, lawyers.

Then I guess you've never dealt with Red Hat.

Sure I can spend a few hours Googling about mysterious log entries, blindly try various solutions that will fuck my system even more, and eventually I might even find a solution.

OR I can just ping my support contact, get on with other tasks, and receive a magical solution by the end of the day. And if it's a software bug, they might even patch it!

Wow that sounds like commercial support actually worth paying for.

I work a lot with MS and we pay them a lot for support, which is terrible. It's double-outsourced (first to Accenture which then outsources to a number of other companies).

It means the support agents are very far from anyone in the core product and they are measured on the amount of escalations so they are extremely reluctant to do that.

Meaning if you have a question that's easily answered in the docs, it's great. If it's not and something is actually broken on their side (which is usually the case because we can read...), you're stuck defending yourself against "It says in the docs it should work so you're doing it wrong". Once you manage to convince them something is really not working as it should they'll stall by constantly demanding logs and updates to the latest version. By the time they review the logs another version is out and they will demand that.

Eventually after a couple weeks you can manage to escalate to an account manager and force the issue to go to someone who actually knows anything. It's the kind of support that really feels like part of the problem rather than a trusted advisor.

I'm kinda surprised RedHat isn't like this, in my experience it's only been small companies that I've seen excellent support from.

I can't help that think this kind of move should be expected once IBM bought red hat. The core of the issue is an trying to extract more value from a well run/well optimize business which usually happens after such large acquisition. There is a fundamental limit to the kind of return on investment that a OSS first/sell support model can provide. In the short term, it's probably more profitable to pivot to more close model, and try to extract even more value from captive audience... But only time will tell how that pans out in the long term.

Is there any risk to the Quarkus project from Red Hat's involvement? It's been my go-to for Java webservices, but Red Hat's track record is making me consider something like Helidon.

Hi. Irrelevant with this thread, but can you please reply to me here if you got the time: https://news.ycombinator.com/item?id=35955336

Search for my username and you will find my question! Thanks!

Respectfully, what's wrong with CentOS for testing purposes?

Isnt 100% compatible with RHEL, so the result won't really represent what would happen on RHEL.

If it's not 100% compatible then how compatible is it?

If 100% compatibility is the requirement then the only option has always been RHEL built, packaged, and distributed by Red Hat. Not some 3rd party's infrastructure.

Apparently the package versions don't match and the order they are released don't match RHEL, so it is not possible to be sure a package that works on CentOS Stream works on any given version of RHEL.

CentOS Stream is not the same operating system as RHEL. Original CentOS nearly was. 99+% of the time, if you ran into a bug in CentOS, that bug could be replicated in RHEL. For most people, that 99%+ compatibility was enough.

Why should Red Hat provide its main product (stability, compatibility, support) for free? Why are enterprise users so entitled? Why should Red Hat care about them at all?

While I agree with that partially, unfortunately I don't agree that it's a wise move for IBM. There is indeed a weird dynamic where IBM is basically funding third party companies to make a profit reselling what is advertised as a bug for bug compatible version of their own software while struggling to sell the original.

So, I can see the logic of IBM wanting to end that. Of course the flip side is that that might prompt those third party companies to cut loose and join forces on creating a proper alternative that is effectively an independent fork that replaces Red Hat as an upstream. Amazon would be well familiar with that playbook of course. And they already have Amazon Linux as their in house preferred Red Hat derivative. And I seriously doubt Amazon cares a lot about bug for bug compatibility. In fact the whole point for them is probably not having most of those bugs to begin with. They just want a stable upstream that they can pull changes from.

That's what makes this such a dumb move. IBM/Red Hat might get a few users to jump ship to the paid Red Hat. But most simply won't. And the further which ever fork emerges as the alternative diverges the less likely it becomes that users will still want to consider switching. Users want a stable upstream but it's not a given that IBM is the ideal steward of that upstream. And with Amazon, Oracle, Rocky, Alma, etc. all needing the same thing, the logical alternative for them would be to unite and form a foundation that takes that role.

This is great actually! Redhat continues to keep the same maintainers and devs on payroll and working on the same open-source projects, so the OSS ecosystem won't loose anything in that regard.

But where will the big research labs that previously used Scientific Linux, then Almalinux go now? CentOS Stream? Debian? They will switch some systems to RHEL, but probably only a minority.

I wonder what distro will gain from this, makes me excited for the future.

I for one would be okay with a distro like AlmaLinux migrating away from being an exact RHEL downstream clone to being an independent stable RPM based distribution. Stability and RPM ecosystem compatibility is what people are really looking for when they use a distribution like RHEL. So any reasonable distribution that provides those two will be sufficient to migrate away from Fedora/Redhat entirely.

I wouldn't, and neither would a lot of other shops I'm familiar with. Being able to run binary compatible distro in dev/test/QA and pretty seamlessly migrate to supported RHEL in prod is an important cost and risk containment measure. Now it's either 1) deal with 'is this bug our issue or because of something in the close-but-not-exact distro', 2) Dump our current supported distro for another; RH support has been in my experience very, very good, and no I don't think Rocky or the rest will come close any time soon, or 3) take a pretty big financial hit licensing RHEL everywhere.

May be time to look at SuSE again.

I'm reminded of Drew Devault's 2021 blog post " Open source means surrendering your monopoly over commercial exploitation":

https://drewdevault.com/2021/01/20/FOSS-is-to-surrender-your...

How about everyone chip in so that Rocky/Alma can get a subscription to RHEL and get the source that way? The GPL does not say they have to publish the code in the wild just that they have to provide it when they distribute. There are many companies that use OSS, make modifications, and since they never distribute are not required to make that source available.

We could make a big list of people that are ready to buy a subscription. Everytime RHEL has an update:

* A subscription is purchased * Download all the source rpm's. * Put them on the internet * RH terminates the license and the purchaser is put on a list * goto next person on list and repeat.

A few thousand people would keep this going.

Seeing how Intel, AMD, AWS, etc. are involved in some way or the other in maintaining their distros or supporting RHEL derivatives, it would make sense for them to band together and do an LTS distro from scratch, or pump enough money into Canonical so that it becomes a viable competitor. The distro is not where the alpha is. So why all the fuss ?

Why on earth would they do that?

Only if they make Canonical dump snap.

Pouring money into canonical seems unlikely to make ubuntu better. Paying people to work on Debian as individual contributors probably would.

This is a natural outgrowth of being owned by IBM. They are in the business of harvesting revenue from existing customers until they revolt. They don’t care about open source or smaller customers. They care about squeezing and auditing the enterprise. This is bad for RedHat because there are other options.

I started my life with a plethora of distros, but I still remember redhat 5.2. Redhat in the latter years as it tried to incentivize became known to me as redhate. Their insistence on gnome over kde was what broke us up, I thought getting bought by IBM would be the nail in the coffin, it appears I was right.

Anectodal data point to provide some perspective to the IBM acquisition. My friend worked Redhat sales towards the end of 2000s. As we all know Redhat makes money from support. Everywhere they went IBM was eating their lunch with their Redhat support sales.

> When Red Hat decided to turn the community CentOS distribution into a leading-edge distro instead of basically "Red Hat Enterprise Linux, but free", users like me were justifiably angered.

Funny sentence there.

Yeah, I don't get it either.

On a related note, folks who are thinking we'll just use Debian or SUSE or whatever other distro, don't understand what RH does. RH is one of the principal contributors to core Linux components used by all distros. Who makes some 10GbE driver work really well? RedHat. And that driver makes it's way back into the kernel sources used by all distros. So I think there's a lack of understanding of the dynamic here.

The allure of RHEL clones is that it's all been checked over by organized engineers who's job security depends on the quality of their work. I would much prefer not to rely on a couple of free-timers producing a result equivalent to what RH does.

And yet there is a huge demand from folks that simply cannot pay what RH wants. So my guess is that something new is going to come out of all of this ....

Just use AL2023 aka Fedora LTS. 2 years of updates and 5 years of maintenance. That is exactly what everyone wants. Long term support and more frequent updates.

IBM, are you d*mb?

Yes.

Is anyone surprised?

Red Hat has been running roughshod over the Linux community for some time now. I say good riddance!

Red Hat is IBM now. That might not be clear today, but give it 5 or 10 years and it will be.

IBM bought them in 2019. Do you need another 5 or 10 years to figure it out?

I don’t, no. I knew the future before the deal was closed. So did most of my friends and former colleagues (who quit at the first sign on the dead hand reaching out into their business units).

It's not because they're not dumb, it's because they're IBM...

This is a bummer. But as a NixOS developer I gotta say it also has its upsides.

Glad that we dumped CentOS/RHEL in favor of Ubuntu LTS years ago.

This isn’t a RedHat problem, it’s an IBM problem.

Of course they're dumb, they are IBM now.

Are you asking why a commercial company doesn't want to give its main product away for free? I'm surprised CentOS lasted as long as it did.

what do you feel RH's main product is?

is it:

1. a collection of very old binaries of software with lots of testing done, or 2. a linux distribution that happens to have a very good support contract and a web of agreements with other vendors

Based on the experience I had with their support (covered by a support contract), it must be 1.

So the value is having to live with the same bugs for 10 years?

The value is not having to reverse-engineer, rewrite and recompile software every week because some random library changed in a non-compatible way.

For some people yes. The bugs you know about usually will have a workaround that minimizes it's impacts. It's the bugs you don't know about that are typically most difficult to deal with.

I guess the question that would lead to is why a company that didn't want to give it's main product away for free would have had a main product that was licensed as open source for so many years.

We all know that an open source license means people can use it, copy it, modify it, and run it where they want, without paying anyone, right?

(Is it even legally possible to have a linux distribution that isn't open source? I'm not sure?)

> Is it even legally possible to have a linux distribution that isn't open source? I'm not sure?

For the GPL parts? No. But that doesn't mean "you must make it the source code publicly available at https://example.com/source-code.tar.gz".

GPL 2 and 3 differ a little bit in the exact terms for this, but both are essentially the same: you only need to ship source code to the people you distribute the software to, not to the general public at large.

You need to ship the source code to the people you distribute the software to and you need to do so under a license that allows them to modify and redistribute this.

So if any of your licensees wants to make a 100% copy of what you shipped, then you need to allow them to.

Kind of, but also no. As I understand it, RedHat customers are forbidden to redistribute the source code by the RedHat EULA. While they can't legally sue you for breaking the GPL, they can sue you or just drop you as a customer for breaching the EULA. I'm not entirely sure how this interacts with the GPL's "no additional restrictions" clause in a legal sense, although it clearly isn't in the spirit of it.

Either way, the net effect is the same: no one is going to risk the wrath of IBM's lawyers, and any avenue of "source leaks" can be closed off. It's not a functional mechanism for RHEL derivates.

I'm not sure that is accurate about the EULA?

The EULA I find googling (this is not something I am previously acquainted with) says:

> With the exception of certain image files identified in Section 1.3 below, the license terms for the components permit User to copy, modify, and redistribute the component, in both source code and binary code forms. This EULA does not limit User's rights under, or grant User rights that supersede, the license terms of any particular component.

— https://www.redhat.com/en/about/licenses/rhds-rhcs-eula

I may not be interpreting that correctly or what it means practically, I'm not really familiar with what's going on here. If you definitely know better than me, I'm happy to hear it (cites to other informed authors explaining it would be great!), but I don't want to go down a long thread accidentally discussing things based on a misunderstanding of the EULA...

I do get that Red Hat is trying to somehow add value to open source software (linux), have it's software be understood as being open source itself (for the marketing value), while also trying to get people to pay to use it. They are trying to have their cake and eat it too, I get it. And maybe they have enough lawyers to make it work somehow, or can just make it so inconvenient practically to use the rights the open source license grants to make it work somehow.

But yeah, if you "don't want to give your main product away for free", and your "main product" is licensed with an open source license... you are in a pretty difficult spot. Which is, I guess, where Red Hat finds themselves.

I've seen it mentioned in a few places; it took me a bit to find it, but it's in the "subscription services agreement"[1], the relevant bit being section 2.2:

"Distributing the Subscription Services (or any portion) to a third party outside the Portal or using the Subscription Services to support a third party without paying the respective fees is a material breach of this Agreement even though the open source license applicable to individual software packages may give you the right to distribute those packages (and this Agreement is not intended to interfere with your rights under those individual licenses)."

I think the way this works (legally) is that accessing their service (their Customer Portal) is covered under this EULA, while the software is covered under the license. So technically you're not adding additional restrictions to the GPL. Or something like that. Not a lawyer etc. etc.

Edit: I see there's also some discussion of this over here, in case you're interested: https://news.ycombinator.com/item?id=36436375

[1]: https://www.redhat.com/rhdc/managed-files/GLOBAL_Cloud_Subsc...

Makes sense -- it sounds like you can't let someone else use your credentials to access the Customer Portal website/online services... or maybe there's something else that's a "subscription service" they're trying to limit too that I don't understand... but they seem to acknowledge you still have the right to redistribute the source code of any GPL licensed code, that the GPL gives you, right?

But yeah, they are trying to figure out how to square the circle and have their cake and eat it too. It does not seem like they are actually trying to tell you that you have lost the legal right to redistribute GPL software though.

The license doesn't actually say that. It says thar if you ship someone a binary then you also need to give them the source on request. Not the same as "everyone has a right to free access to the source".

Open source licenses "actually" also say that anyone you ship the binary and source to is free to share the source with whomever else they want to.

So an open source license to me seems to be in conflict with the idea that you "don't want to give it away for free", but if you think a license that requires you to give users the source and then lets them distribute that source as they will is a different thing than "giving it away for free", I don't know if I'm interested in arguing about it. Although I guess I'm curious if that's your position.

The "four freedoms" are "the freedom to run, copy, distribute, study, change and improve the software". Note "copy" and "distribute". Any user of your software is free to copy and distribute it.

"Open source" has for decades _not_ simply meant "you share the source code with your licensed users", while not allowing modification or redistribution except by further license. There has for decades been a difference between "open source" and simply sharing the source with paying customers while not allowing them to share it with third parties -- and attempts to rewrite history and make "open source" mean nothing more than the latter are attempts to rewrite history.

You are right that making software available under open source license does not require the original owner/author to run any particular infrastructure of distribution though. It does not require them to run a web site or git repo or anything else. It just says that anyone with the software is allowed to share that software with whomever they like however they like, as well as make modifications and share those modifications with whomever they like, as well as run that software however they like, with no further permission or fee required because the open source license already grants permission.

So, I stand by my comment -- if you "don't want to give your main product away for free", and you license your "main product" under an open source license -- you have made a terrible mistake. (Or are engaged in some kind of calculated deception of your userbase). Because the open source license has indeed already granted the world the right to use your software for free.

The thing is, I'm not so sure RedHat actually lost revenue over CentOS. The people who want to use CentOS is a very different type of userbase than the people who would use RedHat/RHEL, and the CentOS people would simply choose Debian or something else over RedHat.

From purely a business perspective I think the entire thing naïvely makes sense, from a far-away view without much knowledge of the business space. But once you look a bit closer I don't think it does.

They definitely lost revenue. People use RHEL/CentOS because a load of software only officially supports Enterprise Linux. But why would they pay for RHEL if CentOS works fine?

> a load of software only officially supports Enterprise Linux

Which software is that?

The database engine IBM DB2 didn't (doesn't?) officially support Centos. It had to be RHEL.

At this point at least they seem to support RHEL, CentOS, SUSE, and Ubuntu. Or at least, they offer downloads for it – I'm having a hard time navigating their documentation to find a more precise answer.

But the people who are running Db2 probably has very little overlap with the people running CentOS, too.

https://www.ibm.com/software/reports/compatibility/clarity-r...

Basically all commercial EDA software. E.g.

https://www.synopsys.com/support/licensing-installation-comp...

Then they shouldn't have poked the bear.

By taking out the CentOS Foundation, which was NEVER going to sell support. They created 2 companies that will sell support, and they still have the same issue.

At the WORST those companies will have to buy a RHEL license somewhere in here, if Red Hat takes away redistribution rights on the srpms... they might as well put a gun to their own head.

SuSE will take the business gladly.

But, sometimes the people that use CentOS get hired by Mega-Corp and when their CTO requires a support contract with contracted SLA's, they choose RHEL instead of Ubuntu One, because they're way more familiar with RHEL'like systems instead of Debian systems.

Yes, cheap/free people will be cheap/free, but sometimes they're not the decider.

The product is open source. They chose to make a linux distribution.

The code and bits aren't Red Hat's main product. Red Hat's main product is the "warm fuzzies" that large enterprises get from knowing that somebody is "on the hook" to provide support if there's a SHTF outage or other incident, and somebody to sue in the unlikely chance that they lose serious $$$ as a result of a technical problem. In many ways Red Hat is closer to a company that sells insurance than a company that sells software.

I disagree. RedHat can do whatever they want just like you can do whatever you want. Don't like what RedHat is doing, then change your distro.

If you want to target their OS just contact them and ask for a license.

Stop with demonizing source code privacy or monetizing our craft.

I did this. I was told to use CentOS or pay for the developer license thing at USD100/year.

Which I did. It's not a lot of money, and it gets me access to support, more docs, etc. I don't really begrudge them that fee, given all the FOSS work they support.

RHE going enshittification?

Can someone explain what the value of RHEL is supposed to be?

Yes, your Red Hat sales person can explain it, or perhaps their web site. Posters are not here to do the bare minimum of research for you that is freely available.

This comment sums it up quite well I think: https://news.ycombinator.com/item?id=36438048

Red Hat is IBM, do the math.

This is not the inflammatory surface level comment that it seems to be, Red Hat invests and supports a lot of opensource projects and there are many good people there, but..

Red Hat is IBM, do the math.

Imagine if a Windows developer complained that they had to get a license for Windows in order to make sure their software worked on Windows. It is entitled to think that just because you are a developer that you deserve a license.

That's a fair point, but as one of the developers who complains about it, the difference for me at least is I don't support Windows or Mac unless someone in the community will test it. I only have Linux in my life, and I'm not buying Windows or Mac in order to support those platforms. I'm not buying RHEL to support that either. I'm a big Fedora user these days so I'll probably end up supporting RHEL by inference since most Fedora packages will work unchanged on RHEL, but I won't be testing it. I'm also considering moving back to Arch, but Fedora is so good that it's hard to picture leaving and Arch has the same problem that Fedora now has (that there's no "server" version I can use, and I like to use the same ecosystem on servers as I do locally. That's actually why I started using EL in the first place and have since been the cause of hundreds of purchased licenses over the years).

And no CentOS Stream and Fedora Server aren't quite good enough because at this point in my life I have many dozens of boxes to maintain. I can't migrate every box every 6 to 12 months when Fedora revs, and I can't install and test updates constantly with CentOS. It would take a ton of my dev time that needs to be used for dev.

Historically, Microsoft had very easy to access developer programs to ensure that people were building software for Windows. More recently, they gave out free VM images for time-bombed Windows for the purpose. Currently, Windows 10 is free if you don’t need to access customizable features like desktop background and are okay with a permanent notice on your desktop.

Agreed.

I'd argue it's never been easier to run a legitimate copy of Windows with very few hassles without paying for it. Sure, there's a watermark, but that's it.

You don't need to buy a license to test software on Windows: https://developer.microsoft.com/en-us/windows/downloads/virt...

Even as a dyed-in-the-wool Open Source Guy, I give kudos to Microsoft for managing to avoid the pitfalls of it's peers. 25 years ago, everyone seemed to hate the company, but they've done a great job of turning a profit while not alienating developers and operations folks.

It's self-serving of me to like this clip by Balmer, but maybe Microsoft gets it some times.

"Developers, Developers, Developers, Developers, [...]"

https://www.youtube.com/watch?v=Vhh_GeBPOhs

I don't see this as being about entitlement either way. RHEL isn't entitled to Jeff testing his software on RHEL, and Jeff isn't entitled to test his software on RHEL for free. The point was that a previously mutually beneficial arrangement is ended.

The thesis is that Red Hat needs 3rd party devs to test on RHEL more than 3rd party devs need access to Red Hat, and if this is true then the move is dumb.

The business hustlers of Hacker News demand everything be given to them for free so they can then extract maximum margin for their Salas built on top of free (as in beer) software. It angers them when they find out that someone needs to pay for OSS development too.

Salas/SaaS. Autocorrect:/

Reply on news.ycombinator.com