Hacker News Reader

Comment by freedomben | Hacker News Reader

Disclaimer: I'm a former Red Hatter but my opinion is my own

Red Hat, you NEED to fix this. I get that it sucks to have your business built on selling support, and a couple of companies have popped up selling cheaper support but not taking on the cost of building the thing. I get that (you think) it's going to cause problems for you (it might, I'm not sure yet). But you're torching an insane amount of good will that was built up over 20 years of good and mostly selfless deeds. You are making yourselves an enemy and a villain, rather than a friend. You need to stop the bleeding.

I don't have any answers, but I have some ideas:

1. Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support. If those don't agree, other projects will emerge who will agree and will take their place. Don't make their lives unnecessarily hard by throwing up roadblocks in their way. There will probably be popup companies selling third party support for Alma/Rocky etc here and there, but I doubt that type of support competition makes a serious dent to enterprise sales. In fact, I think it helps you by increasing adoption. A rising tide raises all boats, especially in this field.

2. Figure out a way to let individuals and small developers use real RHEL without having to dick with subscriptions or licensing or activation. Be liberal with this and make it easy

View on HN

I would like to add the perspective of a small-business infrastructure guy here.

First things first: we're a Rocky Linux shop. About 100 instances, some bare metal, some VMs. A few local, a few at cloud providers like DigitalOcean. Held together by Ansible and kickstart installs.

Too often the discussion about RHEL clones like Rocky paints a picture of people like me as freeloaders. And some of the folks with a similar setup might be. But most small-time Linux admins I know are real open-source believers. I would really like to funnel some money into the platform we're using. First and foremost because it's the right thing to do, the people writing all that source code deserve it. But also because we need platform continuity, and being able to open a support case from time to time would also be great.

But it's really hard. I don't have the time and nerves to deal with the RHEL sales department, deal with the licensing, produce my own cloud images because DigitalOcean can't provide them, deal with our own internal purchasing process, deal with convincing my superiors that we need to.

Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

We're the same, but a little larger. Don't let anyone convince you you're a freeloader. If you're a freeloader, then Red Hat are freeloaders for relying on the Linux Kernel and all the open source software in their distros as well.

What we both are are power users that don't really need their support. Our company has an entitlement so we can use the KB articles (to our benefit) and view/submit bug reports to them (which is to their benefit). I have no problems telling them that (and for the most part did last week), and if someone on the call were to argue me on that point, I'd have happily gotten into it with them. I know they're adding a lot to linux and open source, and pay for a lot of developers to maintain and add to projects (including Linux itself), but that's just the point, they're adding to something that already exists that they use. They don't own it, and they can't expect to require payment for it. They aren't Microsoft, and the only reason they've ever been able to compete in that space is because they're standing on the shoulders of what the open source community has made possible.

It's a symbiotic relationship, and they can mess with it at their peril.

> then Red Hat are freeloaders for relying on the Linux Kernel

"freeloaders"

Most Active Linux 6.1 Employers [1]

By Changeset

[...]

7. Red Hat 672 4.8%

By Lines Changed

[...]

10. Red Hat 24073 3.1%

Most active employers, 5.16 through 6.1

By changesets

3. Red Hat 4916 5.7% (behind Google and "Unknown")

By lines changed

8. Red Hat 202698 3.3%

[1] https://lwn.net/Articles/915435/

and that's just the kernel. Red Hat also runs or contributes to a bunch of other projects that are significant, especially relating to containers.

Not just containers, all the core linux stuff. gcc, binutils, gnome, systemd, etc.

One of the major differences between buying a support contract with RH and buying one from literally any other vendor is that RH doesn't sell support for things without having someone active in that project's community.

AKA, there is a good chance the gatekeeper that your !RH support provider is talking to is a RH employee or funded by RH. Put another way, rocky can't afford to hire 1k engineers to maintain all those packages, nor can anyone else so your trusting that the bug you want fixed can be fixed by a drive by contributor. This is part of the reason that RHEL has such a small package list too, just about every single package in that list has someone working upstream.

So, you won't get stuck like i did 15 years ago with a glibc bug that the company your paying a support contract to can't fix. Sure that distro patched their own version, but the change got dropped a couple years later during a rebase, and it was the only distro with the fix. Meaning our product only ran on that specific distro + version until we reopened the bug with RH and they actually got it fixed upstream.

To be clear, you're less likely to get stuck like that. There's plenty of bugs open that Red Hat doesn't fix because it's either too problematic or affects too few people. Sometimes that's eventually solved with a point release and them rebasing to a newer package version which includes an upstream fix, sometimes it just persists. I'm not saying it's common, but it's happened, and happened to us. Encountering a bug and finding the KB article with only workarounds and then a bugzilla entry that's been open for a couple years isn't exactly common, but I also wouldn't count it as all that rare in my experience.

Also to be clear, I wasn't making a case abovc that Red Hat doesn't provide a lot of benefit to the open source community, just that Red Hat could not exist without that same community. I do not begrudge Red Hat making money. I do think they're being extremely shortsighted if and when they shut off the larger open source community's uses of their core OS product. They're making their whole ecosystem much harder to get talent for if they do that, and it's not like entities running Rocky or Alma are just going to spring for RHEL. We trade on our staff's skill as System/Linux Administrators, and the core OS is not as important as all that. It's like if Python started requiring licensing fees. You'd get some short term return on that and some larger entities will just pay, but you'd see python usage in the general community plummet, with all that entails about its future.

Maybe to their eyes it makes more sense in this current world of startups and VC to just exclude the smaller players, since those companies that grow organically to a size where Red Hat's pricing makes sense are much rarer?

>Not just containers, all the core linux stuff. gcc, binutils, gnome, systemd, etc.

Depending on who you're talking to, this may not be a good thing.

... and looks like they're locking many of those changes to the source code behind their own paywall.

While it may technically be legal (ie crafted by lawyers to make sure of it), that still seems like a pretty shitty thing to do.

Red hat upstreams a lot of their patches, much more than Debian or Ubuntu.

I'm aware, I used to work for Red Hat (2010-2015). :)

There's also different types of contributions. Hardware vendors like Intel and AMD are mostly contributing their own hardware drivers (and a lot of that, especially in AMD's case, is just autogenerated hardware definitions), whereas Red Hat is mostly contributing to core kernel functionality.

> If you're a freeloader...

It's dishonest to leave that conditional off the quote. The work and risk/stress involved in running a supportless shop is quite high. So if they are freeloading by using Rocky then Red Hat is also freeloading.

I do think it's funny that they weren't even number 1 on that list though. Guess they are at least getting a lot of volunteer work from other companies!

The others are almost all hardware manufacturers doing new hardware enablement.

It's hard to tell without more context of your own whether you're supporting what I said, or refuting the argument that Red Hat are freeloaders, which is an argument I did not make.

Thanks for the feedback!

We've just about finished transitioning to 100% Rocky, coming from a 50/50 setup with many of the bare metal installs being Ubuntu. But Ubuntu left a real bitter taste in my mouth with the snap enforcement (I can't even start this snap thing in my Ubuntu Docker container? And you took away the native apt? Wow.), so we started a migration to 100% Rocky.

Telling my manager that we'll have to work something out after we just about finished working something out will be my own peril as well. Let's see where this will end.

I would wait a bit to see how this shakes out before raising the alarm. I'm waiting to see what Alma/Rocky do, as well as holding out a small amount of hope that RH and others will agree to something.

Yeah, we were late on CentOS 8, and had just put 3-4 systems and VMs into production when the Stream announcement hit, which made us stop in our tracks. We eventually accepted Rocky and started actually getting newer systems in place once 9 was out, so we just jumped to using 9 and in-place upgraded our Rocky 8 systems to 9.

We're only about a quarter migrated though, so still have a few hundred Centos 7 boxes to eventually get off of, and we'll see if that ends up being Rocky or something else.

I will note that this episode of The FOSS Pod[1] with a Debian lead actually made me think switching to Debian itself way back in the past (since we've been doing this for almost three decades now) would have been a safer path. I highly recommend it, it cleared up a few misconceptions I had as a Red Hat/CentOS user about what Debian is really attempting and why, and how that would fit with our needs. For example why their goal of pure open source licenses isn't an ideological choice, but a choice to benefit the Debian ecosystem, since the goal is to make downstream distros of Debian easy and without problem (e.g. no extra legal review needed based on licensed included).

1: https://fosspod.content.town/episodes/debian-linux-with-jona...

Same here, we still have 2 or 3 CentOS 8 to Rocky conversions to go from before we even started replacing Ubuntu!

I have to admit, while I share a lot of the values of Debian, I never really liked the product. Especially the apt package mangler, err, manager. They've also carried some Debian in-house patches that really caused me some grief in the past.

I've started running FreeBSD on my personal servers. Unfortunately it's not an option at work.

I've never really used Debian in the past, so can't comment on the reality, just what I've heard, and even my Ubuntu experience is lacking. I'm running it as a virtual desktop now, but I try to keep my use of it to an MUA, web browser and terminals so I don't have to spend the time to become familiar with it, given it's got no other use for me so far.

Our director loves FreeBSD, and we actually use it for firewalls, so it's not a spectacularly hard sell. I'm not sure it's significantly different from the problems of Stream to make it worthwhile, at least not without some external patching and support entity in addition, but it's been a long time since I looked at the specifics of their packaging and ports systems. I don't actually mind something like Stream for a personal server actually. It's just when trying to manage a fleet in the tens and hundreds when it becomes a real liability and increased possible admin support load IMO.

Asking from curiosity, what is the problem you had with apt? I'm using Debian close to 20 years now, and liked Debian and .deb a lot, more then RPM even.

Honestly, having built some packages and maintaining a derivative provided me more insight than a regular user, but even before that I was using Debian without any problems.

BTW, Ubuntu is not Debian with a different UI. They're pretty different on fundamental things, even.

A broken pre/post install script (often the problem is not expecting the current state of your machine) that fails while running `apt-get install -f` it the common culprit.

Just curious, why didn't you migrate from Ubuntu to Debian? Wouldn't that have been a smaller change?

Because we already had a solid base of RockyLinux VMs, so standardizing made sense. And I personally dislike apt / Debian.

Did you really compare our freeloading shop against one of the biggest software contributors to Linux and supporting ecosystem? RH is a top 5 contributor of patches across the most important core components.

Red Hat adds to the open source projects, they didn't write them from scratch (in most cases). Using freely provided software and building your business on it is perfectly acceptable, whether it's us using Red Hat's sources or Red Hat using the Linux Kernel sources and hundreds to thousands of other open source projects. We extend the OS where it doesn't fit our needs, and submit patches and feedback and bugs to both Red Hat through a subscription for a system we have for just that purpose, as well as to the upstream projects RHEL is based on, just as Red Hat contributes to the upstream projects.

So yes, I made that comparison. If you think a company using a RHEL derivative product is freeloading, I'm not sure how you can believe Red Hat isn't also freeloading on the open source community, but feel free to explain you position.

I, for one, think neither is freeloading, but just making use of the open source community to provide extra value for ourselves and our customers.

> Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

I think they do this on AWS and Azure (and maybe GCE?). DO may just not see enough of their enterprise customers asking for it for them to dispatch the lawyers and negotiators to make this happen.

Disclosure: Ex-Red Hatter, but never got anywhere close to anything related to sales or licensing.

Thanks for the hint! I will keep it in mind. Still waiting for the situation to really calm down. We'd rather not spend precious time and effort on changing clouds, and even if we did we'd still need to figure out the situation regarding bare metal and container images.

Most likely switching from DO to AWS or Azure is going to greatly increase your costs

>Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

We've tried to make deals with RH in the past, but they will only let us sell RHEL VMs with licenses if it's running on a RHEL hypervisor... which none of our platform is using at all.

I think this sucks, but for a little bit of defense of RH on this the entire support stack is built around assuming a RHEL host (meaning you're using a supported and exact version of the hypervisor and associated tools). Adding support capabilities for a non-RHEL host would be a major task, and not one they are well equipped to do.

Now that said it's a bit dumb because with most things it's not hard for support to tell your customer, "yep that's a problem in the guest" and help, or "that's a problem with the host, contact DO" or whoever.

But in general, when Red Hat says something is "supported" they are guaranteeing a lot about it. They'll literally fix the code if something is wrong. There's also a culture internally of "be careful helping with something that is 'unsupported'" because customers have previously accused RH people of messing stuff up and demanding they fix it. I helped with stuff like that most of the time when I worked for RH, but I was always very clear with the customer that "this isn't Red Hat, this is me helping as a friend and fellow Linux nerd" and the rapport with the customers was always good enough that I wasn't worried.

As someone who has done this, and benefitted from others doing this, I think the open sharing is probably the number one reason to use these types of solutions. You get people that are in their own niche and know specific parts of the system inside and out. That builds credibility and increased profits, but the companies are too nervous to stand behind their people.

> you're using a supported and exact version of the hypervisor and associated tools

Having exact versions is not a problem (any version of RHEL guests are supported on any RHEL hosts past and future, almost). Plus Windows, ESX, Amazon and Google clouds, and a few more.

Rather, the problem is having _up to date_ versions; with RHEL you can often ask the customer to reproduce on an up to date host if possible, with other distros the support machinery isn't there and you need someone to talk to at the cloud provider, so that they can debug stuff that breaks only in their environment and can't be reproduced. Otherwise you cannot guarantee the level of support that you mention. Very few cloud provider can provide that, I am not even sure that IBM's cloud made the short list.

Isn't long term support Red Hat's thing? The ask isn't to support some random system, it's for RH to put out an official image and distro they support on cloud providers and bake the licensing costs into the hourly runtime fees.

This is exactly how you consume Windows on AWS and it's a huge incentive to do it that way because it takes licences management out of consideration.

The cloud provider is also running software, the cloud doesn't run on magic dust. Red Hat doesn't supports RHEL on your blood if they don't believe they can trust you to help supporting RHEL, the way their customers expect. The easiest way is to run RHEL as the bare metal OS but it's not the only way.

> This is exactly how you consume Windows on AWS and it's a huge incentive to do it that way because it takes licences management out of consideration.

It's also how you consume RHEL on AWS, GCE or Azure.

Blood -> cloud of course.

That's sad.

And we'd have to figure out what to do with the bare metal instances and containers, too.

"Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing."

This is a two-way street. It's not all on Red Hat. I don't know to what extent Red Hat has or hasn't tried to do a deal with Digital Ocean but it's not as simple as Red Hat just deciding to do it. There's a fair amount of certification work that DO would have to do as well as hammering out the legal / business agreements.

The "comparable" part exists with many cloud providers, so if you're not married to Digital Ocean then that part should be doable. Red Hat also has some programs around subscription portability.

However, I do get it that the subscription process introduces friction and for small shops that have a patchwork infrastructure as you describe, it's going to be difficult. Some of that is on Red Hat to solve if they want your money, but not all of it. They can't do it all themselves by fiat if Digital Ocean isn't willing to meet them halfway.

We're not married to DigitalOcean, but changing cloud providers also takes time and investment we'd rather spend elsewhere. And we'd still have to figure out what to do with the bare metal instances, container images and whatnot. The friction is very real.

you have to understand, why would digital ocean want to deal with this when they can use actual open source options without the overhead?

Because they might some day find themselves in a tight spot if all of those open source options have become a half-broken wasteland because nobody invests time/money.

being afraid of all free linux distros going away because no one invests in open source is some epic level FUD.

How exactly do you think those companies even came to exist again?

From your description, IMHO you're not a freeloader, especially since you want to contribute. You are the exact type of people that I think Red Hat is harming themselves with.

I'm in basically the same boat, currently using Linode. At other companies I've made decisions to buy RHEL and OpenShift, and would again because I've been pretty happy (we'll see how this current thing shakes out).

I also went to work for Red Hat (and took a pay and title cut) largely because I had so much love for them. I'd been using Fedora and CentOS (pre-stream) for many years. The expertise that I gained from the experience also let me hit the ground running in my new job, and I made them a lot of money. Had CentOS not existed, I would never have learned their ecosystem and certainly wouldn't have gone to work there. That's profit that is hard to measure, and it is hard to directly tie it to somebody's commission or KPIs, but I'm sure it's not unsubstantial. Many of the people I worked with had a similar backstory!

I would also say (at least when I worked there) I didn't know anybody (who wasn't a sales person) who would have called us free loaders. Quite the opposite, they considered us part of the community.

This is essentially my backstory as well. I'm typing this on a Thinkpad running Fedora. RedHat seems to be focused on big corp and government customers. But even they are not an island. Loosing the grassroots will really hurt them in the long term I think.

Honestly, there are a ton of people like you out there. A big selling point for Red Hat has always been their contributions to the community and that funding them would help to drive that.

It is a big motivator for people transitioning from your state to becoming a customer.

> Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

Most of the major clouds have this pre-existing capability to charge more and revenue share the extra through their marketplace services etc; and they're in active use by other vendors. It's often a good source of revenue for both the cloud company and the vendor, and helps the customers by providing them with prebuild solutions that they don't have to build themselves (e.g. web application firewall vendors can/do provide closed source images that customers can use that way). It's not like this is expecting cloud companies to build whole new services from whole cloth just to support Red Hat.

>Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

This 100% already exists, but I don't believe Digital Ocean is one of the supported vendors.

You with 100 instances is pretty sizeable. On the other hand, I know research infrastructures (read: thousands of instances around Europe) built with CentOS 7, Alma & Rocky.

These guys are also power users and developers who need no support.

This is a massive blow by RedHat, and I'm not sure that they're aware of the size of the install base they are attacking to.

> Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

This exists, but probably only for the larger cloud providers and not e.g., DigitalOcean: https://azuremarketplace.microsoft.com/en-GB/marketplace/app...

I had a meeting with our RH reps last week, as a followup to Red Hat Summit and to talk about a potential purchase with them. I had a discussion about how we use free RHEL derivatives, and have for almost 30 years (when Red Hat itself was free), and we might have used one of the first public releases (I do know we were running 5.2 in '99).

In explaining our position I told them we use the free products (which meant CentOS until stream, and now Rocky), I noted how I'm sure they have their own reasons for what they did, but the Stream change made us seriously consider other distros like Ubuntu, and while we settled on Rocky in the end after it came out, if we had chosen Ubuntu we wouldn't be having a discussion about possibly spending $100k with them now and more in the future, so that's how they're hurting themselves.

Then again, from attending Red Hat Summit, it's apparent that they're really aiming at the enterprise and government anyway. I would estimate attendance was 60% government and education (including a lot of government contractors), 30% enterprise (Adobe, etc), and 10% small to medium business (and I'd bet many of them are partners offering services and want to support Red Hat).

To me it looks like people making short sighted moved to increase their hold on their enterprise customers (this might be aimed more at Oracle Linux than Rocky or Alma), but at the expense of the surrounding ecosystem. The knock-on effects might be that they kill the ramp to RHEL and the training of admins in its use except for those companies willing to actually shell out for official training programs, as there will be little to no regular use by non-enterprise customers anymore if they keep this up.

>> there will be little to no regular use by non-enterprise customers anymore if they keep this up.

They are slowing transforming RHEL to become the new AIX (https://en.wikipedia.org/wiki/IBM_AIX).

What is that line? “No one ever got fired for buying IBM”?

I think the main appeal for enterprises is that if their VM’s start going down and thousands flush through the drain every second, someone at Red Hat is there to pick up the phone and hop on a call with their engineers. It makes them and their regulators happy and no other Linux distro offers that. I don’t think they care about open source at all.

> What is that line? “No one ever got fired for buying IBM”?

It's a wildly common saying in enterprise, and it basically means, "if you choose a startup or unknown product, it might go really well or it might be a bomb that gets you fired. If you choose IBM, you'll spend more, get less of a product with less quality, but you'll never get fired for it because IBM is a 'safe choice'"

> I think the main appeal for enterprises is that if their VM’s start going down and thousands flush through the drain every second, someone at Red Hat is there to pick up the phone and hop on a call with their engineers. It makes them and their regulators happy and no other Linux distro offers that. I don’t think they care about open source at all.

Correct, enterprises don't care at all about open source. This is IMHO a bit part of why Red Hat is stopping caring as much.

Now that said, when I was a consultant with Red Hat I worked with a ton of RH customers and there were individual engineers who did care. Not nearly as many as I hoped, but there were at least some.

I think this is a key part of where they've been aiming for years. Since the '00s, even. Maybe earlier, IDK. I just have their public behavior to go on, but it sure looks like a long-term strategy. If it's not, it ought to be, because they're doing most of the work for it anyway.

1) Steer enough of the Linux ecosystem that Red Hat is the most-qualified support source for common Linux stacks, including maneuvering to make it difficult to decouple or avoid them in other distros (Systemd, Gnome, and Wayland are key plays here).

2) Cut off access to the blessed least-risky-for-enterprise version of your product.

3) You are Linux for much of the market. You own it. Profit.

At the (current) top levels, you might be right. I don't think it's been going on longer than 10 years though. I'd guess maybe 5 or 6.

The bulk of engineering doesn't like this, but sadly for a while now it's been a sales-driven company, and the IBM influence has certainly not helped. Sales doesn't give a shit about the health of Linux, or even the long-term health of Red Hat. They want their commission and promotion, and pesky open-source ideological things get in the way.

How about since July 9th, 2019?

What happened then?

That's when IBM bought RedHat.

I would say at least since udev was merged into systemd and gnome adding an hard dependecy on systemd-logind [2], both in 2012 so I would say more than 10 years.

To me that this was an act to force a Redhat-led project through other Redhat-led projects.

[1] https://lwn.net/Articles/490413/ [2] https://gitlab.gnome.org/GNOME/gnome-settings-daemon/-/commi...

Gnome and Wayland are the last worries for any enterprise customer.

Gnome's part of the play to make it painful for other distros to avoid Systemd, which does concern ~all enterprise customers. And I'm sure some of those enterprise customers are paying for graphical workstations, besides, even if it's a minority.

[EDIT] If I'm not communicating this well, consider: if Red Hat steers several key projects that basically all other serious distros are stuck with (in no small part due to Red Hat maneuvering that sure looks like it was designed to bring this situation about on purpose), how can another vendor ever be anything more than a knock-off of Red Hat? Ubuntu tried to fight them on each of these points, and others, and lost every time. Result: if you're running Ubuntu, much of your system is defined by, and the future development of it steered by, Red Hat. You're practically running a Red Hat knock-off, aside from their dead-man-walking FlatPack competitor, Snap. May as well just pay for support from a budget RHEL vendor... oh, wait, those just got cut off, guess you'll be paying Red Hat. It dramatically increases their moat and the cost of building a credible competitor that's not basically just off-brand Red Hat with less ability to steer the future of the platform.

systemd makes enterprise customer's lives easier. It's more reliable and it's easier to write your own units and timers than what came before.

Systemd's a lot more than an init system, and that's no accident. Red Hat now determines where Linux goes in some very important ways, and I think they believe they've finally got a solid lock on that, after years of effort, if they're making this move. Why go with a competitor (if you're a paying customer, which is who matters) that's just following in Red Hat's wake? And now, the options for RHEL-in-a-different-label will be gone. Who's left? Nobody, you're gonna pay Red Hat or eat extra (perceived, if not actual) risk.

[EDIT] Nb that for the above it doesn't even matter if Systemd's better than the pile of interchangeable daemons and programs it replaced. I have opinions on that, but it's irrelevant to Red Hat's market maneuvers, aside from that it was important to them that it not be easy & safe to cleanly replace, in part or whole.

Have you looked at 2022/23 commit logs for systemd and matched those authors to employers?

(Preview: It's mostly Microsoft, Meta and Suse.)

I think that's why this current issue is more relevant than ever. Red Hat is burning goodwill they had, which is going to eventually start to cut into the types of decisions they used to be able to make when they had more community "power". I've always enjoyed using Debian (or Debian-based) systems than Red Hat (or Red Hat-based) systems, and was unhappy when I had to re-learn init scripts, but at this point in time Systemd just seems to be easier with some features being "free" to restart or figure out systems that don't restart automatically.

The funny thing is that it looks like most of the decisions leading to that outcome have been overwhelmingly coincidental.

I would be very surprised if it's a coincidence that Systemd's as difficult to avoid, if you're building a Linux distro, as it now is, nor a coincidence that it's been going NIH all over the rest of the system ever since Gnome was leveraged to speed up its adoption.

The point is to make it difficult to offer a credible enterprise Linux distro that's not full of Red Hat-steered software in key places, turning you into a cheap knock-off of Red Hat with little say in what happens to the software you're providing, in terms of future development. It now costs a lot more money to get out of Red Hat's shadow & control, as a would-be Red Hat competitor—they're not just another distro among many, they run the damn show.

You missed 'deprecating the alternatives' :)

Somewhat jaded opinion after many years in the business, but these companies exploiting OSS for profit have learnt from how Microsoft's moves have played out over the years (and are now making their own mistakes). Given the only way they make money is by providing support, obviously they're going to go the MS route (make operation arcane and complex, as per your first point).

> You missed 'deprecating the alternatives'

I don't want to disagree too hard with this, because I, too, have been uncomfortable with the march toward making it very difficult to build a Linux system without things like systemd (I actually don't have all that much of a problem with systemd anymore, but I also value choice).

But no one company can deprecate anything. The source is all there, under permissive licenses. If people want sysvinit or openrc (etc.) to work well in order to build systems without systemd (for example), they are free to maintain, support, and evangelize them. And this happens, to some degree. If it's not happening to the degree we're all happy with, then we should pitch in to help, either with our labor, or financially. Otherwise we don't really get a say.

They just have to make it hard/risky/expensive to keep using things they've declared deprecated, or to avoid the software they're pushing & steering. Which they've done pretty damn effectively (again, if this were all somehow accidentally, holy shit, it shouldn't have been, because, wow, it's been a highly-successful-and-effective set of business maneuvers executed over a span of years)

Doesn't matter if some hobbyists still run Gentoo with openrc and X-Window with Windowmaker and all the classic interchangeable daemons and components that Systemd replaced (and with which it is incompatible)—it matters that any commercial competitor's only realist option, short of a massive and risky investment, is offering the same things Red Hat does, but with less influence on those projects. They want it to be expensive to build a competitor that's not a Red Hat knockoff—and if that's not what they wanted, well, somehow they took a whole bunch of steps over a span of years to make it happen anyway, by accident I suppose, but the outcome's the same.

> Doesn't matter if some hobbyists still run Gentoo with openrc and X-Window with Windowmaker and all the classic interchangeable daemons and components that Systemd replaced (and with which it is incompatible)

Don't forget alpine is also openrc and it is far more used by corporate tech than gentoo. Especially as a base for containers.

As ever, solidarity matters. The only way to get out of these embrace, extend, extinguish cycles is to form independent organizations that are never beholden to a single company or even a small handful. Individuals can't stand up to the efforts of teams of company-paid developers just by contributing their free time. But it's very hard to create such an organization without the companies immediately recognizing the need to stuff as many of their proxies into said org to gain influence, and that's a difficult thing to counter. As an ecosystem, we need to contend not only with whether a particular thing like systemd works well or solves a problem but also whether it has predatory intentions. This is simply what the incentives lead to unless the community has a strong voice to counter such attempts. And in this day and age, that also includes dealing with all the submarine articles and other consent manufacturing that are well within the budget of any large company to try to fake organic community discourse.

That sounds a lot more like Canonical's (attempted) strategies

1) Be the defacto "new user" or "newbie friendly" distro for two decades

2) Keep making asinine attempts to monetize your users (Amazon search bar comes to mind)

3) Try to pivot and break core Linux functionality so software only works on you (Keeping Upstart after everyone else abandoned it, Mir instead of Wayland and the horrendous amount of forking done to GTK and others to get that hideous 'Unity' desktop going)

Agreed, I think Canonical has been a lot worse in this respect than Red Hat. I think Red Hat's technical arguments have merit, and they do a pretty good job separating projects from the company so they are definitely not RPM distro-only.

Yes, Canonical has been their main competitor on each point and it's possible the threat of Canonical is what sent them this direction in the first place [EDIT: or vice-versa—Ubuntu may have attempted to "own" parts of their system in response to Red Hat's moves to do the same, I'm not sure how exactly that timeline works out and all I have to go on is how both have behaved].

Canonical-steered alternatives to Systemd, Gnome, and Wayland, have been advanced (as you note: Mir, Unity, Upstart) with less political and/or technical savvy than Red Hat's efforts, and have all failed. Technically they all still exist, in some form, but they're de facto dead. Snap/FlatPack is another place they're butting heads, and I think it's fair to say Snap's not got a bright future, but I guess it's technically still in the running, so we'll have to call that one undecided.

How exactly would Red Hat pursue such strategy and compete with Canonical? Consider the fact that Red Hat leaves the copyright to the contributor in Red Hat's projects, while Canonical requires contributors to sign CLA to get patches included in Canonical's projects. Given this arrangement, Canonical both benefits from the contributions and also has extra rights over the code that everyone else doesn't have that gives them a competitive advantage.

What's that got to do with maneuvering for control of extremely-important parts of Linux? If anything, license/attribution fuckery would put you at a disadvantage in that kind of a game. Red Hat was playing for position, not material, in chess terms. Canonical may have held more rights to their competing software, but it doesn't matter a bit, because those projects are irrelevant now. They "own" more, perhaps, but they're in checkmate, so it doesn't matter. Who's looking to pay the best-positioned claimant to "upstart maintainer" for support? Practically nobody, and that's not a growing market.

If you're an enterprise, do you see Canonical/Ubuntu or Red Hat as the most-credible vendor to support Systemd, which both of them are running? Is there anyone else who might hold that title? The answer to those questions are the point of what Red Hat's done (or else they've accidentally achieved a pretty amazing coup through a series of maneuvers that led to that outcome simply by chance, which notion strikes me as unlikely).

What's important is making it extremely expensive to launch or re-position a competitor distro to Red Hat that actually ships substantially-different software than Red Hat does, at a level of polish & reliability that might let you compete with them. It leaves competing distros as Red Hat followers. Which, fine if some hobbyists pick that, what they care about is being perceived as the only credible claimant to the title of first-party vendor of Linux, in most contexts by enterprise.

> I think it's fair to say Snap's not got a bright future, but I guess it's technically still in the running, so we'll have to call that one undecided.

I'm surprised they haven't thrown in the towel yet at this point.

When I read HN, I never see anything positive about snap. And while this is arguably a fringe community, it is very highly correlated with the 1% of influencers to corporate decision makers.

I think you can really only use this assumption if you are talking about desktop Linux. Using the Ubuntu server version is mostly giving you Debian with more packages available (and all the benefit and downsides to that).

> Keeping Upstart after everyone else abandoned it...

False. Ubuntu switched to systemd as soon as its upstream (Debian) did.

See https://www.markshuttleworth.com/archives/1316

Not false :)

Ubuntu offered Upstart starting in 6.10 (2006) through 15.04 (2015) after Debian's (admittedly very politicized) switch to systemd forced their hand

RHEL 6 (2010) included it for a single release before switching to systemd in RHEL 7 (2014) (though in balance, systemd was started by a Red Hat employee)

No other distros switched to it, and the ones that offered it as an option were very second-class citizens

So, they were one year after RHEL on switching to systemd to the default (and it was available as on option before that). The first LTS version of Ubuntu released after RHEL7 had systemd as the default. I wouldn't really say that they were using it "after everyone else abandoned it". I'd characterize it more as they developed an alternative init system that never really caught on.

Indeed. And notably, upstart came long before systemd, and with it Ubuntu was the first major (or any?) distro to solve the real problem of the non-parallelized boot speed issue for desktop users. Characterising it as "pivot and break core Linux functionality" is just outright misleading.

a.k.a. The more things change, the more they stay the same.

Few of us have careers long enough to remember purchasing SCO or HP-UX or similar and what that whole experience was like. We're positively blessed compared to how things were in the '90s.

Red Hat's leadership forgets what made them successful in the first place and wants those bad old days back badly.

When you're not at the top you want competition, since you are competition. When you're at the top you want to choke competition and gate as much as you can. It's a story as old as trade itself.

"Systemd, Gnome, and Wayland are key plays here"

GNOME and Wayland are utterly non-strategic for most of Red Hat's business. If your theory of Red Hat's long-term strategy hinges on those being strategic, then you need to go back to the drawing board. You could fairly cite systemd, I suppose, but Wayland and GNOME not so much.

On point 3 - that's not really controversial. All the Linux vendors have tried to "be" Linux in the public market. Canonical has certainly tried to be the brand-name Linux for most users.

To the extent Red Hat has tried to steer the market w/r/t its tech stack, it's done so pretty honestly and by trying to persuade others to adopt its tech stack and by doing the work in the open. This is the opposite approach of Canonical which has tried to push lots of NIH technologies to silo people to Ubuntu and continues to fail at it over and over again.

Gnome and Wayland countered Ubuntu's attempts to "own" competitor projects in those areas (the current fight is Snap/Flatpack). Gnome, in particular, was important as a lever used to make it expensive for distros to avoid Systemd, forcing it into de facto standard status more quickly than it may have achieved on its own merits (if it ever would have—and maybe it would!), from which point they've started using Systemd to eat other system services & daemons, increasing their control over the platform and cementing their status as, effectively, the "first party" vendor for Linux itself—not literally the kernel, but enough of the system that you've got to go pretty far into here be dragons territory, by the standards of enterprise, at least, not to be using Red Hat-steered software in key parts of your system—so if you want a say in the direction of Linux, as you experience it, or priority bugfixes from the best-possible source, or to talk to experts who write that software for a living, for advice or training, you're short on options other than... Red Hat. Which is the whole point.

Gnome is a healthy non-profit foundation with the majority of its organizational capacity (and income) not associated with Red Hat.

Red Hat hasn't been dominant in Wayland for years. It's very much a community effort, including governance and maintainership.

systemd is currently mostly developed by employees of Microsoft, Meta and Suse, aside from I guess Frantisek.

Red Hat does a lot, and I'm sure they have thoughts about where they spend their money, but I don't think people really understand how decoupled these projects and communities are.

It's funny how people view GNOME/systemd as some sort of conspiracy rather than it just being what the people who contribute want to work on.

Corporate strategy isn't conspiracy. I think it's even weirder to think redhat would just provide resources in return of nothing at all, or that redhat employees won't be biased towards steering the ship according to their employers best interest.

Like I genuinely don't think Gnome or systemd or Fedora would just use a ubuntu/canonical-built core system even if it was superior in every way

And hey I'm not complaining, I use systemd (indirectly) and gnome almost everyday. I don't care where the software comes from, really, I'm just glad I can use it. But that doesn't mean we can't be aware of the power dynamics around it :)

What about Upstart?

It was significantly better than sysvinit and Debian, Fedora, RHEL6, OpenSUSE all adopted it. Even though it came out of Canonical.

It was replaced by Systemd because that did concurrency in the dependency management, which Upstart didn't, and was never designed to do, so was going to require a lot of rework to support.

> the current fight is Snap/Flatpack

I work with enterprise customers every day who use RHEL, it's free variants, and Ubuntu - I can promise that none of them give a single shit about Flatpak or Snap. They will never use either of them. Those tools are only relevant to Linux on the desktop - the small size of that market is exactly why both companies (mostly Canonical) have gotten away with being a nuisance on this front.

Flatpak is very relevant on the immutable OS space which is being pushed hard as the future. It is mostly an enterprise (server/vm) thing too.

I just don't see people caring about an immutable OS in the enterprise space. The central requirement is already addressed by containers, and all of the security and compliance stuff enterprises do is predicated on having a "normal" OS - even if an immutable deployment is technically more secure, that doesn't help you if your auditors can't understand whether it checks the requisite boxes.

Yeah. To me, this just seems like IBM feeling like their existing (corporate) sales funnel is big enough that the consequence of this (reducing OSS users based sales funnel) aren't going to meaningfully matter.

Or, at least they won't matter while the current decision makers are still at IBM. It could be a problem for future IBM leaders, who will probably just attempt to buy SuSE or Canonical or something at that point. ;)

GNOME was pretty instrumental in forcing systemd to becoming a GNU/Linux-wide standard though, by creating some dependencies there.

> This is the opposite approach of Canonical which has tried to push lots of NIH technologies to silo people to Ubuntu and continues to fail at it over and over again.

Agreed, Canonical is worse. The whole closed snap store (where only canonical can use it) is an obvious play on milking the commercial software market.

That doesn't mean that what RH is doing is great though.

> Systemd, Gnome, and Wayland are key plays here

Systemd, sure, but I would assume that the vast majority of Red Hat's support contracts would be for server use, where GNOME and Wayland are irrelevant.

Gnome was a major lever in making it hard for other distros to avoid systemd, early on. The way Wayland's been structured and that... transition, such as it is, has been managed, smells a whole lot like a fire-and-motion move, and besides, dominance there was necessary to keep Mir from winning as the successor to X-Window and making it possible for Ubuntu to follow Red Hat's playbook and use it to push some of their other, deeper-in-the-stack projects (they'd have been neatly positioned to interfere with both Systemd and Gnome, had they won). I don't think Red Hat would let Ubuntu push anything of note, at this point, without countering it, for (justified!) fear of their own tactics being turned on them.

https://www.joelonsoftware.com/2002/01/06/fire-and-motion/

Systemd's the main body of Red Hat's army here, if you will—no doubt that's the most important part of all this—but other projects have played other important roles in supporting or defending it, at times.

It's not like Red Hat planned and schemed to execute on Wayland as a plan for domination, but their decision to support Wayland as an existing project with technical merit staffed by skilled people might well have been tactical for the reason you mention.

Some people tend to imagine the first situation and rightly dismiss it as a post facto conspiration but the second situation is utterly realistic. They may even not have realized until much later how right they got.

And I say that as someone who is convinced we need something like Wayland, the sooner the better. But we also have to avoid being naive about how business is done.

>You are Linux for much of the market. You own it. Profit.

What a twist!

So after all it wasn't Microsoft EEEing Linux, but actual "Linux Companies"?!

Microsoft tried and failed because their product is strictly inferior for the market which Linux has ended up dominating. Marketing and evil behaviour can definitely win out over technical quality, but we were fortunate that it didn't go that way in this case.

I'm not sure if that argument holds water. You have Azure which is hugely successful, and Linux is still not anywhere close to becoming a standard desktop environment beyond the type of people you find on HN. Microsoft contributes quite a bit to Linux now, because they see the future. Becoming a leader in open source gives you the ability to steer where technology goes, and Red Hat definitely was able to play that hand as well (but under IBM seems to be losing quickly).

I'm not sure that any of your post actually contradicts mine; MS have accepted the dominance of Linux in the server market and have moved on to exploiting it where they can. That doesn't mean they wouldn't have rather had everyone running on Windows, if they could have pulled it off.

Also former Red Hatter, disclaimer also applies.

According to my observations, Red Hat does an unbelievable amount of stuff by hand. Lots of internal processes require someone to do something at a given point, otherwise these distros are not happening.

I believe RH is now cutting costs at precisely these places: projects that don't directly contribute to the revenue get cut. However, as the post said, these changes make it incredibly hard to ship open source software for RHEL-likes to the point where I already decided to look for alternatives when doing integration work.

> Red Hat, you NEED to fix this

You should be addressing IBM. This is how they operate. Things will only get more strict and more difficult to work with from here. They don't care about good will. They don't care about OSS. The only thing that matters is dominance in the market place to take care of their bottom line.

A lot of Red Hat people pay attention to forums like this, and I would bet money there's a big discussion going on internally right now. IBM couldn't care less, but the people at Red Hat will. They're also not powerless (at least, they weren't when I left a few years ago).

There were similar discussions when the CentOS Stream thing was announced and Red Hat didn't change course so I'd guess they're not going to back off this change either.

> There were similar discussions when the CentOS Stream thing was announced and Red Hat didn't change course so I'd guess they're not going to back off this change either.

True, and it's likely nothing will change this time too. But if there's no outrage at all and nobody tries, we guarantee nothing changes and probably invite further encroachment.

Obviously IBM owns Red Hat now and all that. But people overestimate how much involvement IBM has day-to-day with Red Hat. (Source: I'm a former Red Hatter, left last year.)

At worst I'd imagine that IBM's influence is felt this way: IBM expects x% growth and other metrics every quarter. How that is achieved is up to Red Hat's executives. If the leadership at Red Hat decides it can do that with the status quo, then I don't think IBM is going to say "no, you must change the source distribution to fit our evil plan."

The worst thing that IBM has done directly (IMO) was plucking Jim Whitehurst away from Red Hat and over to IBM, but not making him CEO. Whatever reason IBM chose to do that, it wasn't good for Red Hat or IBM long-term in my opinion.

> Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support.

That was CentOS (after acquisition by Red Hat but before CentOS Stream). Unfortunately killed off.

Agreed, although there is one important difference: Red Hat isn't behind Alma or Rocky like they were CentOS (yes they did that to themselves by acquiring CentOS, but that was in the past. It was a mistake, but we have to move forward from where we are today). A lot of people knew that CentOS was owned by Red Hat and that made them a lot more comfortable using it. They kind of had to change that relationship somehow. I'm not happy at all that they did it, but I can understand why.

Option 1 above allows them to keep that "those are third parties, not Red Hat" relationship but still have the benefits of a free community build.

I worked for a company that did the voting for American Idol for the first few seasons. Weird culture, a mix of bare metal and NIH that I thought didn’t quite function and some of us were trying to change that culture. Everyone resented working on that project and wanted to do other things, so at some point we disinvited ourselves (or so the story went, might have been sour grapes).

The problem was we killed the cash cow without any viable livestock lined up to replace it, and the layoffs started about a year later. Of course 4/5ths of the people trying to change the culture were in the first round. Get rid of the rabble rousers first.

While I think our proposed changes would have helped dramatically, I also think it might have been better if the company management had invested in making it more palatable to work on the project. Bribery if you have to. Conferences, toys… but they didn’t do either, and it flopped.

Edit: on rereading your comments I may have missed your angle, but I’ll leave this anyway, since we are talking dumb failure modes.

That I can’t run sudo yum update out of the box with RHEL is one of the big reasons I don’t bother with it. I love the “stability aspect” of RHEL but I love the I guess “convenience” of any other distro being ready to go out of the box, no activation required

I totally agree. Even when I worked at Red Hat and could get whatever licenses I needed, I still used CentOS for local stuff and for testing things because of the friction of the registration/activation process. It has gotten easier in recent years, but simply existing is enough to turn off a lot of people.

"Me too" - Ex Red Hat. Development is done on Fedora and CentOS. I was pissed when they changed CentOS mainly because it changed my workflow.

Since RHEL 8.2 its at least possible to register at installation time.

Alma, Rocky, fair enough.

But isn't the elephant in the room here Oracle Linux?!

Yes, very fair point. Oracle and Amazon both deserve to be on the list, probably much further up than Alma or Rocky

Amazon actually seems to have seen this coming. While Amazon Linux 1&2 was a basterdized CentOS6& derivative (with some newer packages), the latest breed (Amazon Linux 2022/2023) is based off Fedora.

Also, I'm not convinced Amazon (or Oracle for that matter) have a very bright future. The opaque development process and (lack of) community support make development pretty hard. I know from experience, getting bugs resolved (or just getting Amazon to care) is very hard. At the same time, even for a large corporate maintaining your own distro brings with it considerable costs.

Wouldn't it be great if instead they would work together, share some of the costs and profits of support and cloud mutually?!

Amazon missed then, the company to bet on is SuSE here.

Both Oracle and Amazon Linux have VERY bright futures for their use-cases.

Do you run Oracle? Want to know your distro will be supported 100%. Oracle Linux seems like a damn good bet to use.

Are you one of the largest companies on earth, who needs a distro to run your business? If so... guess what.. your distro is viable just on your own business.

Amazon offering Amazon Linux is just them showing some internal tech. Nothing more.

Amazon (and Microsoft) offering a Linux distribution is about (a) raining in the explosion of distros their various teams depend on that they don't themselves control, and (b) answering insistent cloud customers requests for "What distro do you recommend? Do you support {distro}? Do you auto patch {distro}? How come update to {distro} broke your service/product? Do you not work with {distro}?

Last year many services on Azure went down because they all used Ubuntu and Ubuntu pushed an update that broke for a DNS configuration that Microsoft uses. Having your own distro won't guard you from fuck-ups of course, but you can imagine the difficult conversation Microsoft had to have with their biggest customers about how Canonical can take down your 40 billion dollar business globally?

> Red Hat, you NEED to fix this.

No, they don't. They don't need to do anything.

> But you're torching an insane amount of good will that was built up over 20 years of good and mostly selfless deeds.

Doesn't matter. That goodwill doesn't really benefit them. How many companies paid Red Hat because they liked the things they did for open-source? Realistically, very few if any.

> Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support.

Realistically, there are lots of companies that would pay for Red Hat but don't because they get can get it for free.

> Realistically, there are lots of companies that would pay for Red Hat but don't because they get can get it for free.

If the free RHEL variants disappeared today I'd wager most of those companies would choose other distributions (Debian, Ubuntu, openSUSE, etc) or making the jump to containerisation or serverless before paying for RHEL. In the end less people would be using their kind of Enterprise Linux, which in turn will result in less conversion to the paid version of it.

> In the end less people would be using their kind of Enterprise Linux, which in turn will result in less conversion to the paid version of it.

Why would anyone want to convert to the paid version if their use case is covered by the free version? Out of generosity?

Sometimes support - for the OS or from applications - is actually useful. I worked at a shop once that ran CentOS for most things but RHEL for the database servers, because the vendor only officially supported that.

Additionally as their needs grows and the free version doesn't meet their needs out of the box and the non-free version does they'll be more apt to buy into it given their previous experience with their products.

So the reason for not using RHEL everywhere is the pricing model? I wonder whether it'd be possible for RH to optimize it so that choosing RHEL 'd still be the best option for them now that CentOS doesn't provide the same level of stability.

> I wonder whether it'd be possible for RH to optimize it so that choosing RHEL 'd still be the best option for them now that CentOS doesn't provide the same level of stability.

What you may not be aware of, or have forgotten, is the free versions provided by other companies that provide package compatibility and similar support (as in updates and security fixes) that CentOS did.

Oracle Linux, Alma Linux, Rocky Linux, and Amazon Linux. Quite possibly even more.

Red Hat will have a hard time undercutting them just by reducing their pricing, which is why they're using tactics like a free number of subscriptions/licenses for small organisations and developers and trying to cut off access to the source code.

In our use-case, needing to handle licensing in any form for additional nodes we scale up in non-production would break our processes. We run RHEL on some core infrastructure with equivalents in our pre-production but CentOS everywhere else to give a ‘close enough but free’ infinitely scalable environment to our devs. As the person in charge of the existing relationship with RHEL, there is no amount of discounting or special deals that would make the extra complexity of registration worth it. We need an unencumbered freely deployable OS for our non-production, and for us the result of this change is likely to be a re-platforming to something like Amazon Linux and a seperate support contract or line item on our AWS relationship, rather than giving RedHat any more money and time.

For us, this means we stop doing business with RedHat.

There are plenty of cases where infrastructure can be split into "critical" and "nice to have".

In HPC, it is reasonably common for people to run RHEL on the scheduler, database and login nodes (totalling maybe a dozen servers) but then use CentOS (now Rocky) for the compute nodes, which might be hundreds of nodes. Those customers simply can't justify the cost of RHEL everywhere, even with more recent changes which make it cheaper than it has been historically in this use-case.

To me it sounds like the pricing model is not optimal. I guess in these situations, they could pay RH "a bit more" in exchange for a legal RHEL license covering those extra nodes. But if there was no such offer then it sounds like a lost opportunity.

(but I'm a complete outsider, I just share my views and I do acknowledge I may be missing something, so feel free to elaborate further)

I think you are right - the problem is that "a bit more" really ought to be a small uplift on the needed licenses, rather than the current inadequate discount on compute nodes (along with hoop-jumping to convince sales people you aren't trying to scam them).

Boeing, up to at least 2016. A large amount of RHEL, and a screamingly HUGE amount of CentOS.

It is usually some combination of needing paid support if updates break things, and also generosity. Trust and generosity define the choice of paid support organization.

> 1. Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support.

I don't know how the support business breaks down for this, but for the large customers, I can't imagine any of the little rebranding projects taking the support business that the IT dept. itself calls. "Nobody ever got fired for buying IBM," and in this case it's literally true again: RH still have some of the best technical experts.

(Imagine being a huge company with a urgent mission-critical kernel problem: you want to be able to get a Red Hat commando team parachuting in, including a Sr. Principal Engineer who specializes in hard debugging tasks in that vast code base. Not have Bob's Discount Distro's support line be lost after you say you already tried rebooting and reinstalling your global operations.)

At the same time, if anyone distro is rebranding RHEL not just as an open source hippie volunteer thing (that a lot of not-ready-to-pay-for-RHEL smaller businesses use), but also trying to make it a support business out of it, then that might be poking the lion. The hippie commune might get ripped apart in a rampage. Which might be what's happening now?

(Credible support options from AWS and maybe Oracle are a different matter.)

> 2. Figure out a way to let individuals and small developers use real RHEL without having to dick with subscriptions or licensing or activation. Be liberal with this and make it easy

Maybe CentOS had been providing this market segmentation with upgrade path to enterprise support, until recently? RHEL is not easy to switch to, and I've had a client that was very capable shop look at switching to RHEL, and nope right back, because they didn't want to learn the new bureaucracy. If they'd been able to use RHEL for free since they were smaller, becoming an RH enterprise support customer as they grew would be easy and a no-brainer.

(Docker and then K8s have shaken this up a bit.)

>I don't know how the support business breaks down for this, but for the large customers, I can't imagine any of the little rebranding projects taking the support business that the IT dept. itself calls. "Nobody ever got fired for buying IBM," and in this case it's literally true again: RH still have some of the best technical experts.

I support a very large telco company, and they've had a massive effort underway to convert all of their RHEL systems to Rocky (preferred) or Ubuntu because of the loss of CentOS as something they could use on non-production systems plus their (the telcos) financial issues making them look for any software licenses they can abandon. They've replaced thousands of RHEL boxes with Rocky boxes already, all public cloud images must use Rocky or Ubuntu, no RHEL allowed.

The use of RHEL is limited to critical systems like those running an Oracle DB only (and even those they are investigation replacements for).

almost 20 years ago when I took a Red Hat Kernel internals class (at Redhat) they basically told us to use CentOS if we wanted to experiment with RHEL. (We were preparing for a transition from the HPUX/Solaris machines.). But it worked, some of our architects did that and we could get the scheduler behaving similar to HPUX. I left that company, but I'm pretty sure they did switch and RedHat got a big contract out of it.

We've come a long way.

International Business Machines¹ should merge with the International Brotherhood of Magicians² to get a better chance of pulling this off.

[1] https://en.wikipedia.org/wiki/IBM [2] https://en.wikipedia.org/wiki/International_Brotherhood_of_M...

Not sure if Red Hat will be able to fix this. This is not Red Hat but IBM. If IBM could fix problems like this, they wouldn't have been declining in the past decade.

Maybe a solution for point 2 could be that the first five licenses or activations are free.

That would help, but I think that only works optimally if they also do number 1. If there is any "activation" process or the like, it adds enough friction to cause problems, especially in a modern CI/CD environment. It's not an easy change by any stretch, but if they don't do something they're going to be the next Oracle.

why even go through all of this when debian is sitting right there, free and reliable.

It's not because they're not dumb, it's because they're IBM...

Reply on news.ycombinator.com