Hacker News Reader

Comment by GuB-42 | Hacker News Reader

It looks like Amin is confusing "open source" with "free" (as in free beer).

He seems to be pissed off that Red Hat discontinued the free CentOS distribution. And instead of paying for Red Hat, it goes to Rocky Linux, a free fork of Red Hat, just like CentOS. It is fine (at least legally), but the paid Red Hat distribution is no less open-source than Rocky Linux, in fact, the latter exists because Red Hat is open-source.

That's why I think this article particularly ironic, it says that Red Hat is not "true" open-source and then made a move that is only possible because Red Hat is open-source. It didn't even go to something like Debian, which is the most "free/libre/open-source" of the major distros, it went to the clone of the distro they criticize.

View on HN

As someone that uses RHEL coz of some of our customers need it, I can get their point.

RHEL might be open source but it is structured (obviously) to sell the support license. You can't "just" install DRBD like on every other distro, you need to get the storage whatever the fuck pack, or get the packages from EPEL. In Debian DRBD module just comes with kernel, need to install it on RHEL etc.

It's annoying to manage even as paid customer and wholly worse experience than just installing Debian.

Open source does not mean that you can just install it. I understand that for majority of open source projects that’s true, but definitions are definitions. May be we should invent better terms like non-commercial free software and commercial free software and set some expectations for the first category.

But still Rocky Linux is possible only because Red Hat provides sources for its distribution. There’s no Rocky Windows.

> It's annoying to manage even as paid customer and wholly worse experience than just installing Debian.

All that is mostly solved with CentOS. I really don't get the people so upset about it.

For some of us, using an older but way more tested and more stable version is equally important. CentOS stream moves the release point to (RHEL+1), which sometimes break things in unexpected ways.

This is why we also migrating to Rocky on such systems.

Why Rocky? Why trust the guy that sold to RHEL once already? Why not Alma that has a stated commitment to never sell to a private corporation? Alma also has a stated turn around time for security patches of 2-3 business days, something I have not seen from Rocky.

If we decide we can’t afford 100% RHEL licensing, Alma was my choice, but I see so many choosing Rocky, and I’m genuinely curious why people are doing that.

He sold to RHEL and then RHEL sold to IBM and then CentOS was changed. I don't think selling to RHEL is the condemning move here.

The change had nothing to do with IBM. I know people view this with skepticism, but it's true.

The primary purpose of CentOS Stream is this: to allow a real community to be built around CentOS, where partners and users can contribute. With CentOS if you encountered a bug your only option was to file it against RHEL (not CentOS), wait for someone at Red Hat to get to it, then wait for RHEL to pick it up and release it. You would be very lucky for that whole process to complete within 6 months and if you're not a customer you probably won't be able to test a hotfix on your own systems.

Whereas with Stream: partners and downstream developers can file issues directly against Stream, and then contribute fixes directly to Stream, where they get QA'd in public and are added directly to the future release of RHEL. So Oracle Linux, Rocky, Alma and anyone else can actually participate in the process and contribute back instead of having to just take whatever RH throws over the wall.

However, the messaging and timelines around all of this were handled... not well, and some of the acrimony around that is reasonable even if the changes make sense.

Wasn't that the purpose of Fedora? An up to date distro where new technologies can tested, community and partner feedback, etc.

In any case, I get the justification for CentOS stream, but changing CentOS 8's EoL from 2029, to Jan 1st 2020 ... after it shipped is definitely a good way to alienate your customers and community.

Fedora is where new major versions of RHEL (think 7, 8, 9) come from. Within a major version of RHEL, there was no public upstream under the previous model, and now that public upstream is CentOS Stream.

Fedora moves fast enough that by the time a RHEL release comes out, Fedora has moved far enough to not be very relevant as an upstream.

With the exception of Rawhide, Fedora also has major versions. It looks like people think CentOS Stream is Rawhide or Debian Sid, which it is not.

People think it’s even remotely close to Fedora because it was initially communicated as an upstream, rolling distro (which Fedora isn’t, anyway, so …), and they never read any of the press or forum posts after day one.

Even if Stream had versions, it’s still only tracking about a minor release ahead of RHEL. If you trusted RHEL’s QE process before, there’s little reason to stop trusting it. I really don’t think much has changed in terms of usage outside of some infrequently used software, but you have to account for that when managing them on a RHEL system, too, so I still don’t really understand the problem.

Also, Red Hat is aware CentOS is their marketing tool. If you like CentOS a lot and need serious support, you hire Red Hat.

Another upside of Alma is it is backed by the CloudLinux guys who have been around for a long time with a solid reputation in the Linux/hosting industry—- and I believe you can also obtain a support package for a relative pittance (obviously it’s not RHELcorp-level support, but if you require/need support, it’s something at least).

We have a couple thousand Alma 8 in production and it’s been smooth sailing.

Either appear to be acceptable choices. Looking at their respective websites, they both make similiar commitments to remaining independent. From what I found, the Rocky website actually makes the commitment in stronger terms than the Alma website.

Same here. Probably Rocky spends too many resources in PR. Twitter, mattermost... Meanwhile, Alma just make new releases and patches, quietly.

Rocky is more visible, I don't see that as a bad thing, not like it spends many resources. It also helps partners decide to support them, in particular Mellanox now Rocky-8.

You can always pin your package versions (or image hash) to whatever is in RHEL. So far, all my CentOS stream boxes have been remarkably stable.

And that's for my pets. The cattle is blissfully unaware of anything different.

Looks like our environments are different. The environment we use is a descendant of Scientific Linux, so we always worked in (RHEL-1) environments. So pinning to RHEL is not helping us, again.

You can have a single RHEL8 reference (not sure how hard is to get the full RHEL BOM) to base your versions from and use the CentOS repos as source for the packages themselves.

My CentOS boxes are pulling straight from 9 stream. I had an issue with Grafana today I’m still figuring out.

Your environment is that important but never considered actually using RHEL (or Ubuntu ESM)?

> sometimes break things in unexpected ways

I have faith on RH to keep CentOS stream to be quite stable now that it runs on its own brand.

I can't understand how people had super faith on CentOS when it was run by a community which even had someone lock up the organization's fund/asset at some point.

Price sensitivity is a thing and I'm quite sure I don't get my money's worth with RHEL prices especially when they made us run around explaining our use case to their business people and them telling us months later they're not interested in the end and 'just buy and use openstack'.

You're talking about a community problem (which most community have at some time and I don't believe for a second that companies don't have these kinds of events or worse) when people are talking about what they liked and actually observed to be good for them in centos. Yes, work should be paid. True also that some things can bring more customers without being profitable by themselves. Yes people are allowed to be angry about a bait and switch.

One can be sad and angry that a service that one used to rely on is discontinued or made very expensive.

People were just free riding a community effort off of a commercial product that costs a lot of money and one can be sad when things stopped the way they wanted but angry?

Besides, do all those people complaining have projects that are very sensitive to stability that CentOS stream is going to ruin? That is quite hard to believe.

Not sure if it was even a bait when people just chose to use it and CentOS never locked you up in using it.

Most of the whining sounds very childish.

People believe that RHEL is rock stable.

People believe that CentOS was the same RHEL for free. Bit to bit. May be with changed logos.

So people believed that they’ll get rock stable system for free by using CentOS. They believed it so much that it hurted RHEL sales.

None of those beliefs were correct. But lost sales were real.

One day I'd like to know whether sales were really hurt or it's like 'we've been downloaded 1 million times our single is 5usd, we could have made 5musd' we'... Well no.

People think that we use CentOS because we’re cheap. The reality is the software we use is certified against it.

We’re a supercomputing center. We’re federated. Everybody is using CentOS. So, we also use CentOS on our pets sometimes because it reduces administration load.

We’re using it for years. We can patch it if we need it. We can submit patches if we need it. We use CentOS as a multinational community, which is completely different from “Hey, look! Free RHEL!”.

We’ll see where we gonna evolve. Stream? Rocky? Alma? It’s will be a collective decision. For some pets we migrated to Rocky, and we’re happy, for now.

Do you mind sharing how you decided to go with Rocky instead of the others?

It was visible, had a simple promise (bug-for-bug compatibility), and a migration script.

Being founded by former CentOS founder also helped. MySQL guys found Maria, CentOS founder started Rocky. That’s OK in my book.

All that was mostly solved with CentOS. Changing from a downstream to an upstream distro changes what it is.

Only if you update everything blindly.

Said another way, "only if you don't meticulously map everything to what's stable".

I'm sure there is an easy way to automate that.

That feature is the essence of why most people used CentOS in the first place though.

You can think of it as RHEL tracks CentOS and has it's point releases spun out of that. There is no reason to think CentOS stream will break more often than RHEL does, because, if that happens, it's Red Hat shooting its own foot.

> He seems to be pissed off that Red Hat discontinued the free CentOS distribution.

Just to clarify, CentOS is not discontinued. Basically all that is discontinued is CentOS "point" releases. There is CentOS stream 9, it gets updates, there will be CentOS stream 10, it will get updates, there will be overlap between CentOS stream 9 and 10 being supported. Sure the CentOS stream won't be exactly equivalent to RHEL point releases, CentOS stream 9 won't be RHEL 9.1 - but RHEL 9.1 will be cut from CentOS stream 9 AFAIU.

Something still exists that is called CentOS, but as an upstream to rather than a downstream from RH, it’s not equivalent or even similar to what CentOS used to be.

To me it is not that critical what the specific organizational structure is that relates to CentOS Stream and it's relation to RHEL. What matters more is how it affects me as someone using it.

CentOS was never RHEL, it was though very close, CentOS Stream is also very close. Whenever I used RHEL or CentOS before CentOS Stream I always updated to the latest versions of packages for the major release version, if I use CentOS stream I will still get updates to the latest versions of packages for the major release version. Maybe I get them a bit earlier than RHEL, but this to me is somewhat irrelevant. If I want reproducibility there are still plenty of ways to get it, including RHEL UBI.

UBI is fine for deployment. But in a shop where development workstations are all rhel, it was nice to know that the package versions available for 7.9 matched for both rhel and centos. It isn't that centos stream has no value, its just it isn't the same value. Now rocky is. Our business already made the switch for anything it was using centos for.

> CentOS was never RHEL, it was though very close, CentOS Stream is also very close.

It was close enough, Stream is almost as close and for some people it isn't close enough when they need bit for bit compatibility for the software they are going to run

What is such a big difference as not to keep using it?

You seem to believe a community supported distro was as good as a corporate supported distro and put such a high faith in it but now somehow it's all corrupt when RH supports it putting it as an upstream?

If your usage is that critical, why not buy RHEL (or Ubuntu extended support) for 10 years support and stop freaking out like your Linux days have come to an end?

I seem to think what? Where did you get all that from? Is there anything else ‘I seem to think’ that you’d like to tell me about?

Stream is inherently less well-tested than the normal stable releases. I'm not aware of any specific bugs that have come up in it, but it get updates before red hat is willing to push those updates to people who have a support contract.

Stream also notably has a much shorter lifecycle, with support for 8 Stream ending in 2024, compared to 2029 for rhel/rocky/alma.

To me the main value of using rhel-based distros is the great reliability and long support life. Stream compromises on both of those compared to what normal centos offered, and rocky and alma currently offer.

I don't think Stream compromises in any significant way on reliability. They test it rigorously. The short support window is a bit of a bummer, though.

That said, I'm running CentOS Stream 9 right now. It's supported until 2027. I generally like to keep my operating systems relatively up to date, so a roughly 5 year support window is adequate for me.

5 years sounds good enough. Ubuntu does that too with LTS unless you go extended 10 years support.

People freaking out because their free candy is no longer as tasty as one could hope, it looks quite funny they have zero willingness to pay for such a long support effort.

The shorter lifecycle is only for 8. CentOS Stream 9 is EOL the same time as RHEL 9 full support page (unless I am reading it wrong)

https://www.centos.org/centos-stream/

You are. The charts are confusing. At the bottom it says full support ends May 31, 2027.

Red Hat made some strange decisions with the way they’re handling Stream vs RHEL, plus the communication was abysmal, and they changed models mid-cycle. Be mad at them for that all you want, the rest was overblown.

This company could have just used Stream 9 if he wanted a free OS. Since he’s virtualizing, he could have bought hypervisor licenses for $4k a piece that allow you to run as many VMs as your hypervisor can handle. He could have used OpenShift for virtualization and container orchestration, and just paid for the OpenShift licensing. If you use OpenShift, you can run all the RHEL VMs and UBIs you’d like at no extra cost. What you never do is pay per core. There isn’t a single licensing model from RH that forces you to do that. If you license individually, each license supports one bare metal machine, or two VMs. This guy is full of shit. He’s just mad everything isn’t $0.

Every package that gets pushed into Stream has gone through the entire QA process for being included in RHEL. While it is of course possible that problems could be noticed later on, it's not all that likely.

And to further add, a lot of the major Red Hat backed projects switched to using Centos steam as the officially supported redhat lineage. Ceph and OpenStack Kolla are both good, major examples.

Which are in large parts RH/IBM open source projects, too

"lot of the major Red Hat backed projects switched to using Centos"

"Which are in large parts RH/IBM open source projects, too"

CERN was a big user of CentOS. After redhat changed CentOS8 EOL, CERN was allowed to use RHEL. Last time I checked, they suggested others in high energy physics community to use Rocky or Alma if RHEL is not an option.

CERN is backing CentOS Stream

https://mobile.twitter.com/passthejoe/status/144693144711837...

Well, I checked again. From CERN Oct 28, 2022 Linux Future Committee meeting [1]:

- Several bugs have been encountered in Stream, which were not present in RHEL.

- Some user communities had previously expressed concerns on the feasibility of using Stream in a production setting. This feeling has also recently been expressed internally by some IT groups.

- CERN is evaluating our original recommendation [0] of utilising CentOS Stream as the default Linux Operating System.

[0] https://linux.web.cern.ch/#update-on-centos-linux-strategy

[1] https://indico.cern.ch/category/13390/

CentOS as ‘a useful distribution for what 95% of what CentOS users were using it for’ has been discontinued. We used to use it, and we just can’t anymore - several years before the promised support period was up…

My reading of this is that Rakuten used to use a combination of RHEL and CentOS. Now that CentOS as a "down stream" equivalent of RHEL is dead (its selling point), they have reassessed their platform and gone with Rocky.

Rocky is "down stream" of RHEL, CentOS 8+ is "up stream" of RHEL (since 2020, and what Fedora has always been).

CIQ sell support contracts for Rocky linux, so rather than having a combination of RHEL (with support) and CentOS. They now just have Rocky with an optional support contract.

Linux is open source, they are aloud to do this, thats the point. They are not leaching. Red Hat, by ending CentOS as a "down stream" equivalent, have brought this on themselves.

There will be a lot of other users of RHEL and CentOS who are going through the same process right now as CentOS 7 support ends next year. We will see more companies making the same choice.

I wonder how many RHEL licenses they actually had, a minimal amount to make support requests I'd guess.

I’d bet that’s accurate. The CEO very clearly did not understand the licensing models because you don’t pay per core for RHEL licenses; this isn’t Oracle. VDC licenses are cheap: MSRP is $4k USD per hypervisor with premium support, and I’m sure you can negotiate a discount at a certain size. You can probably game VDC licensing if you really wanted (I’m not endorsing the idea of stealing from Red Hat, and to be clear, my company isn’t interested in getting sued to save what amounts to pocket change for them). Or you could switch to OpenShift for virtualization (because who isn’t at least _considering_ dumping VMWare now that Broadcom owns them?) and container management, and run all the RHEL VMs and UBIs your heart desires.

When people say they want "true opensource", they usually mean they want to be the bosses and dictate how the product is being developed to the people who are actually doing all the work and giving all the funding.

Did you read the article?

Let me quote the article: “I called him and… said ‘listen, I want to give you a shot and I'm willing to work with you’. So we worked with him for 13 months to build our real-time kernel to meet complex workloads with one condition: It has to be open source and it has to derive its characteristics from a large community,”

Rakuten worked with the founder of Rocky Linux for 13 months to develop real-time kernel. I don't know what kind of deal Rakuten promised to Rocky Linux (founder) and its community, but Rakuten contribute by open-sourced the work to community. That's enough for me. That's the spirit of open source.

Is rocky Linux kernel-rt is rhel kernel-rt ? If so i wonder how much work went into the kernel-rt.

It reads to me as though they outsourced their performance tuning. There are tons of tweaks you make to it for specific environments. I’m not sure what they would have done that any experienced engineer could not have done.

>it says that Red Hat is not "true" open-source

This is true though as RH will terminate your contract if you redistribute their binaries. So they use a contractual backdoor to undermine GPL. And before people talk about how GPL covers source but not binaries, it covers binaries. In fact, all of CentOS, Rocky, Alma, etc. is a massive waste of everyone's time, energy, land, and electricity. Something akin to crypto mining. Everyone could have used RHEL, but we need this complicated indirection for no reason.

It covers binaries all right, the way I understand this RH is very aggressively pursues this as a trademark violation (which it shouldn't be unless it's deployed to third party that can give the blame to RH).

The fact is that most users of open source care more about not paying than freedom or openness. The same factor is why people go for “free” surveillanceware or fishing-hook-ware.

The people who actually understand and care about freedom is a small niche.

He is not a stupid person, he knows what he's talking about. Of course he doesn't wan to pay for RHEL licenses, and he explains why the licensing model used by RH now is a bad choice for a business with a high number of nodes.

Which is why proper open source should be GPL, with a commercial license, so that the likes of him pay for the work of others that make his business possible to start with.

Which is already happening. RHEL is open source because the majority of Linux packages are GPL licensed.

If they were using non-copyleft licenses then RHEL packages would be closed.

So long as he’s not making changes and distributing them the choice of open source license is mostly irrelevant.

Proper open source uses whatever licence the author wants.

If the authors want to get paid, they need to choose a license wisely.

That more or less rules out GPL.

That is the whole point of having GPL with an additional comercial license.

Tell that to the MySQL folks. They sold it to Sun for a billion dollars, then made Larry Ellison pay for its development when he finally realised buying it with Sun wouldn’t allow him to kill it.

How ?

I’m still waiting red hat to pay me for the accepted ansible patch I wrote few years ago.

Did you get a contract for work done?

It was a way to reply “has to pay for work of others” but which work ? And which others ? Cause currently Redhat is owner of Ansible and make money with it, while me as a contributor never got any penny they get from Ansible.

So why rakuten should pay Redhat for work not getting paid by Redhat? See the point ? Who is parasiting the ecosystem here ?

He isn’t stupid, but he’s either uninformed, or he’s lying because he’s dead wrong about their licensing model. You don’t pay per core. You pay per physical machine (of any size), or per 2 VMs (of any size). If you’re heavily virtualizing, you buy a VDC license at an MSRP of $4k USD, and you run all the VMs you want on a single hypervisor. If you’ve got more than a couple hundred machines, the VDC licensing is probably your best option. Alternatively, use RH’s virtualization solution, and pay only for the OpenShift licenses.

He doesn’t want to pay for the OS at all, he just wants to pay for support. That’s fine, but he’s still factually incorrect in what he’s said.

I would agree 100%, but this is not 90s Red Hat, it's IBM now.

It seems like the classical "Here's our new pricing and for being a long time customer we throw in a bottle of vaseline too". Which technically has nothing to do with open source but Red Hat ( like every company in this space ) created an "ecosystem" and uses the word "open source" VERY liberally when selling you stuff. Some of their opensource software need some paid service or "maintenance" contract to work or have any value.

I'm sure Red Hat/IBM is ( mostly ) covered on the legal side, but the sausage is made in a very different way.

> uses the word "open source" VERY liberally when selling you stuff.

RedHat has consistently been the most aggressive supporter of Open Source out of any for-profit companies. Not only do they contribute to open source projects, they consistently release their own code/projects as open source early on, which could easily be kept proprietary (if they chose) and used as a competitive advantage over others.

Like what? You’re paying for engineering, which provides stability, and support. There’s nothing from Red Hat that isn’t upstream for free. They even open source companies that they purchase (like 3scale). The base OS has multiple clones (Alma, Rocky, etc.), Satellite comes from Foreman, AAP2 comes from Ansible and AWX, IdM from FreeIPA, OpenShift from OKD, and the list goes on. The only thing I can think of that doesn’t have an upstream equivalent Insights, but that’s because it uses RedHat’s ML systems, and connecting your systems to theirs costs money. Insights-core is open source, but may not provide much value since you can’t connect to RH’s backend.

Reply on news.ycombinator.com