Hacker News Reader

Comment by awill | Hacker News Reader

First, they should have announced this when they killed CentOS.

Second, this is only of value to the teeny tiny companies and individual users with a single server in their home. Small companies with any expectation of growth should probably just use Rocky instead from the get go.

I have a single server in my home running CentOS 7. I gain nothing by using RHEL over Rocky in the future, except with RHEL I have to accept ToS, do some subscription checks, and run the risk that Red Hat cancels this programme down the road. I only see this of value for someone who actually runs real RHEL in prod, and needs to test/validate hardware/software. For anyone just wanting 'free' RHEL, they should just use Rocky.

View on HN

I manage a "tiny company", but we have 10-ish production CentOS 7/8 servers and this sounds interesting. Although my web-app use case is probably fine with Stream, I don't know why everyone is assuming Stream will be completely broken and useless.

It's not that the expectation is that it will be broken and useless. It's that the reason to choose CentOS was for the fact that it has slow release cycles which allow for long support periods.

> I don't know why everyone is assuming Stream will be completely broken and useless.

It probably wont be completely broken and useless, but they replaced a stable OS with a beta, which is particularly galling to many users who picked that OS specifically because it was so stable.

Curious, do you consider Fedora to be an alpha? Did you consider RHEL to be a beta for CentOS previously?

Just trying to see how you define "beta"

I think you're digging in the wrong place, and focusing on the label. This shouldn't be kicking fedora down. It has a place, as does stream.

The point is that with CentOS you expect a super stable release. Red Hat replaced that with what is clearly a less stable project. And as was said before, if you used CentOS, it's probably precisely because you want something super stable with long term support. Those are not the sorts of users who'll want to use Stream.

To me, Fedora is a desktop environment (my preferred one!) and absolutely an alpha/early-beta server option at best.

CentOS Stream seems like "Fedora, again" and does not do what CentOS did. And now, the willingness to blow up CentOS with so little warning means there is a massive trust deficit here.

There's a pretty big difference. While I'm very much not an expert on the RHEL QE process, my understanding is that Fedora is pretty much its own thing that is used to rebase RHEL on major revs. Whereas Stream is RHEL builds that have passed CI testing in the RHEL dev/test process.

> Curious, do you consider Fedora to be an alpha?

Slightly detached because of their processing/release process (Fedora X never quite matched to RHEL Y even if there was some derivation), but yes, Fedora is absolutely the alpha for RHEL. It gets changes, including breaking changes, on a fast track (IIRC, Fedora frequently goes toe-to-toe with Arch for having the most recent packages), has a 13-month lifecycle with a strong expectation that you'll `dnf system-upgrade` regularly, and doesn't shy away from pushing on tech that they believe to be desirable even if it's not (yet) widely used (Wayland, BTRFS, CGroupsV2).

> Did you consider RHEL to be a beta for CentOS previously?

Now that's an interesting point that actually makes me stop and consider:) It is true that previously CentOS could legitimately claim to be even more stable/slow-moving than RHEL. I think the distinction that I would make is that RHEL had its own betas that were called betas, and that each point release was frozen/stable until the next one came along, while Stream is explicitly unfrozen all the time. So yes, I suppose a person could have described RHEL as CentOS's beta, although probably only jokingly because it still had Red Hat supporting it and feature freezes to control changes.

Funny to read this because Fedora is definitely beta, because their distribution doesn't seem very stable. I switched away from Fedora because it seemed to break more than my Arch installation. There are a lot of things they are doing that no other distros are doing right now. Cgroups v2, for instance.

The benefit of Fedora was that they handled a lot of things that were a pain to configure. The biggest culprit being fonts. But now that TrueType patents have expired, and it's included in Arch automatically it's one less reason to use Fedora.

Many people use CentOS to run "Enterprise" software that expects to be run on RHEL.

Those are probably the users that RedHat is trying to nudge onto RHEL. "Enterprise" software is normally quite expensive, so the marginal cost of some RHEL Server licenses is probably a small fraction of the total cost of deploying the software in most cases.

More generally, orgs that are running "Enterprise" software are already set up to pay for software. Going from spending $X to spending $X+Y is often not a big deal ("Ok, I'll run that through the same process I used to get approval for buying FooSoftWorks"), while going from $0 to $Y can be very hard ("I don't even know who I'd need to ask to get purchase approval for that").

> Going from spending $X to spending $X+Y is often not a big deal

Except when $Y == $1000X. Then it really is a big deal.

When you only have a tiny minority of RHEL hosts that you have full support contracts on, and an overwhelming majority of CentOS hosts that you administer in-house, it really isn't trivial.

It's almost certainly cheaper to migrate to Oracle or Rocky. Migrations aren't cheap either.

> More generally, orgs that are running "Enterprise" software are already set up to pay for software.

Crucially, though, most entities who are running their own "enterprise" software deployments will need way more than 16 servers.

Small shops do use "enterprise" software, of course, but they usually just pay somebody else to run it for them.

Okay but I can't help but think about the social security in my country which I know runs likely to be more than hundreds of CentOS servers ... It's not like social security is a company making any profits, it's even the opposite, and now they are in such situation.

We are a smallish company with six CentOS 8 servers. Seems we fit into the profile of this new Developer program. To be honest, I can see us paying for full RHEL subscriptions before we would change distributions.

Go ahead and buy licenses as a show of support, but I strongly suggest that you exercise the converter scripts in both directions, partially to know what will cause them to fail, and partially to have an option when the vendor misbehaves.

Red Hat offers a "no support" server license for $350/year, which is the lowest licensing tier. Tiers with more capability and support are available for $800, and $1300.

https://www.redhat.com/en/store/red-hat-enterprise-linux-ser...

Oracle has a larger range of tiers. The "no support" license is $120/year. Tiers with more capability and support are available for $500, $1200, $1400, and $2300.

https://www.oracle.com/linux/

Oracle Linux can be used in production without a paid license of any kind; Red Hat Linux cannot be used in this way for large deployments (excepting the new 16-seat license for a developer account).

Red Hat is aggressive with software audits; I have seen one.

Both Oracle and Red Hat now have complete toolsets to convert support between an installed CentOS/RedHat/Oracle OS.

Red Hat can now convert an installed CentOS or Oracle Linux to RHEL; previously a wipe and reinstall was required ("have fun reinstalling your system" is still on Oracle's CentOS site). The description looks much more thorough in replacing all possible packages with Red Hat versions:

https://access.redhat.com/articles/2360841

Oracle does not replace CentOS or RedHat RPMs in their conversion (AFAIK), so it is much less violent on the platform changes.

https://github.com/oracle/centos2ol

https://linux.oracle.com/switch/centos/

https://blogs.oracle.com/linux/reasons-for-switching-centos-...

"Red Hat can now convert an installed CentOS or Oracle Linux to RHEL"

Apologies if a dumb question, but I assumed CentOS 8 was a clone of RHEL 8. What would change?

When running "yum update" the packages are pulled from the registered repository, found in the /etc/yum.repos.d directory.

When switching the provider of package updates, the old repository must be disabled, and a new one substituted, among other changes that the new provider requires to switch support.

Oracle makes (by default) modest changes for such a switch. Red Hat essentially rewrites your whole operating system, and has added a warning that consultants should be retained for such conversions in case of trouble.

Take a look at Oracle's "CentOS to Oracle Linux" (and other similar) scripts to see how they work and what they do (sorry, I don't have a link handy as I'm on a mobile device at the moment).

If memory serves, the process consists of, basically, installing the <distro>-release package for the "new" distro (which includes the files in the /etc/yum.repos.d/ (or whatever) directory that defines the package repositories), enabling the repositories it just added, disabling the "old" distribution's repositories, then running a "yum update" to replace the "old" distributions packages (including the kernel) with the (nearly identical) ones from the "new" distribution. After all that is done, you reboot the host so that the new kernel is loaded and that's pretty much it.

Mostly out of curiosity, I tried out Oracle's "centos2ol.sh" (or something like that) script ~8 years ago on a new CentOS 6 VM I had set up for just that purpose and it was as easy as they claim that it is. I don't recall encountering any issues and I suspect that, by now, they've ran into most of the potential issues and have devised ways to resolve them automatically.

Note that while CentOS 8 is a clone of RHEL 8, the CentOS packages are rebuilt versions of the RHEL ones. Most of them are built from the exact same source code but some packages do have minor changes made to them before they are rebuilt (for trademarks, etc.).

It's easy enough to suggest "just use Rocky" but Rocky doesn't exist beyond a collective promise. What about production now, or in the next few months?

> What about production now, or in the next few months?

Pay for RHEL, use CentOS 7, or switch to Ubuntu.

Or scrap all three and start using *BSD.

This seems more and more to be the reality we are moving towards. Been dabbling with FreeBSD lately and it just works. Also, ports are updated or new submitted quite often making it feel less out of date.

That's not a decision you make and then train staff + switch in a short time. At least not beyond trivial deployments. People used CentOS so they don't have to deal with this and they will have other less-extreme options before CentOS EOL.

Even though it sounds bad, Oracle Linux works surprisingly well

It's not just that it sounds bad, you'd have to be willing to sell your soul to the devil.

As I remarked to a RH employee when the CentOS 8 discontinuation came out - do you have any idea how much trust you have to burn that people are considering Oracle as the good option?

RH would have minded. IBM doesn't. They may even take being compared to Oracle as a compliment.

I have heard my share of "at least we aren't Oracle" while I was at IBM :)

That reminds me of Animal Farm.

What's the specific difference between Oracle and IBM/RedHat in this regard? As far as I can tell Oracle has burned a lot of people because they don't do license checks in their software (no DRM of any kind), which is usually what the HN crowd wants. But the flip side is they have to check you in some other way, leading to audits, and audits are very painful.

Is the difference that RHEL does subscription checks so they don't need to audit customers so much?

In terms of open source contributions, RH certainly do more, but Oracle are no slouch. They're doing actually exciting stuff with GraalVM and I don't really remember the last time I saw Red Hat do anything describable as research, their Java stewardship has been pretty responsible, they publish quite a lot of useful tools and stuff.

This page has some interesting details: https://linux.oracle.com/switch/centos/

Whats wrong with Centos Stream?

... you know, I had actually forgotten about that as an actual option. Yes, some people would probably be fine running Stream. I personally would avoid it because you're sacrificing stability by beta-testing Red Hat's software for them, but it is free, it is close to CentOS 8 (RIP), and it is likely to be good enough™ for many uses.

Are there any actual reports of instability of Stream? I mean, it's _barely_ ahead of RHEL/odl CentOS.

I believe CentOS images are still available and supported as much as they were before through the rest of this year, fwiw.

There is also AlmaLinux [0], expected to be available in Q1 2021.

[0] https://almalinux.org/

Agreed. But Rocky is supposed to launch before CentOS 8 is EOL. There should be time to wait and migrate.

You could say the same about CentOS 8. There is no contract guaranteeing support-just a collective promise that was broken in a single day.

At the end of the day, you are looking for a free distro and this is what you can get. If you want a guarantee, pay for a contract. You have the right to sue if it is broken.

Edit: grammar.

(RH Employee)

> First, they should have announced this when they killed CentOS.

I agree.

>Second, this is only of value to the teeny tiny companies and individual users with a single server in their home. Small companies with any expectation of growth should probably just use Rocky instead from the get go.

Without saying too much - stay tuned for further announcements.

yeah, stop playing games, redhat? just say what you're going to say already. i have no idea why you're trying to focus so hard on building mindshare, building a mystery around it like you're a sarah mclachlan tribute band, but you already blew it. go back to what you're good at, please.

I understand as an RH employee you're likely not in a position to make decisions about what you can disclose when, but... (I'm sure you're aware) your first answer contradicts your second.

There's no contradiction, only an acknowledgement that (in my personal opinion) the piecemeal way these plans are being announced is understandably frustrating, but that they are legitimately serious about making RHEL easier to use across a spectrum of use cases (which the official post is pretty clear about).

It's worse than frustrating, it's damaging for Red Hat.

It doesn't matter much to me anymore, I've already migrated my small business internal server that was running Centos 7 to Debian. We had basically just a fileserver, FTP box, and a couple VMs for some time clock, wiki, and building security systems, so nothing we were running yet was CentOS specific.

But in case it can be useful to you, please share with your team that the writing they're putting on the wall looks pretty bad. Knowing I'd have to update eventually, I just took the safe option, which is no longer any distro controlled by Red Hat. I didn't want to leave our server in a state where we could be building dependencies on a platform that was likely to be pulled out from under us, and while RHEL isn't too expensive to just buy for one server, and we might have qualified for something free or something you've not yet announced, my test was surprisingly painless and the migration was easier than I anticipated.

I Agree 100%

RedHat seems to forget the for the majority of CentOS users STABLITY was the driving factor in the choice to use CentOS.

I am not sure what internal metrics they have, or if I am missing a glaring use case somewhere but from where I sit RedHat seems to completely miss the boat as to who is using CentOS in the real world.

I want STABLE UNCHANGING SECURE systems, not the latest and greatest features, or some new shiny "cloudy" technology...

And yet you're not willing to pay for it.

I dont drive those decisions butI will say if they had better "no support" options my org probably would buy it. We have no desire for RHEL support, they had had bad experiences in the past which was one of the driving reason to drop RHEL for CentOS, why pay for support if it sucks?

That's fine, but it would be negligent of a company to wait and hope that Red Hat pulls a rabbit out of a hat. The plan for most companies is going to be a new distro, and they're not going to reconsider when RH announces something new because by then decisions have been made and trust has been lost.

My guess is that at least 10% of the CentOS yum repository traffic was immediately lost at the announcement, as it switched to Oracle.

When your house is on fire, and you have a very short time to save your prized possessions, haste easily leads to observable contradictory actions.

Red Hat was not expecting the reaction to killing CentOS.

The need to expand the developer program into production seats could not have been anticipated.

I don't know of any other vendor that wraps developer accounts into production licenses.

Notice that this access is in no way promised to the end of support for RHEL 8.

I will say again, Red Hat has terminated two major Linux platforms over the last two decades - the original Red Hat Linux, which was terminated at v9, and now CentOS.

Oracle has "somewhat" terminated a Linux platform, Oracle Linux for SPARC, which was only supported for two minor releases. Oracle Linux for ARM64 and AMD64/x86 seem reasonably healthy.

There is other baggage in and between these corporations, but the track record on Linux OS platforms between these two is demonstrably different.

The free converters between them should be used fluidly.

> The need to expand the developer program into production seats could not have been anticipated.

I don't follow. Are you suggesting that RH actually didn't realize that people were running CentOS at scale in production?

I see the decision to convert to CentOS stream as driven by two motivations, cost containment, and facilitating outside contributions by pushing CentOS towards Fedora (and away from RHEL stability).

I might be misreading these motivations.

With this management perspective, the CentOS community reaction to the retraction of the end of life date, and potential reduction in stability, was likely not anticipated.

> pushing CentOS towards Fedora (and away from RHEL stability)

CentOS streams is literally RHEL with fixed major version and rolling minor versions. So, in the past, you got CentOS 7.1, then 7.2, etc., with streams, you get CentOS 8.x.

Saying that this is moving away from RHEL stability sounds highly disingenuous. Are you saying you've seen RHEL 7.X to break as opposed to 7.(X-1) version, and your solution was to wait for RHEL 7.(X+1)?

Is a "preview" as stable as the base release?

Are we not reading this correctly? If not, could you get Red Hat's CTO to retract this [mis]information?

'Another official part of it is, as [Chris] Wright [RedHat CTO] said, is that CentOS Stream as a "rolling preview" of what's next in RHEL, both in terms of kernels and features can be used in today's containerized, cloud-native IT world.'

https://www.zdnet.com/article/why-red-hat-dumped-centos-for-...

> CentOS streams is literally RHEL with fixed major version and rolling minor versions. So, in the past, you got CentOS 7.1, then 7.2, etc., with streams, you get CentOS 8.x.

> Saying that this is moving away from RHEL stability sounds highly disingenuous.

If it were so great, they'd do the same for paying customers. That they don't move RHEL to being just 8.X makes me think that Red Hat does in fact understand that that undermines the stability of the platform.

Red Hat was not expecting the reaction to killing CentOS.

It’s funny how everyone still says Red Hat when we mean IBM.

> Without saying too much - stay tuned for further announcements.

It's a horrible way to treat users.

They aren't paying customers, so whatever for RH short-term bottom line. But amount of community backslash approach like that is bringing can have negative long term impact.

RH pulled the rug from under the people, and is telling them to stay tuned. I wouldn't base my infra on a system like that.

But that's good for overall Linux community - RH has way too much power there.

they didn't have this when they announced the death of centos. this is what they scrambled to come up with when they realized how much of an "oops" move messing with centos really was.

as a peace offering to prevent mindshare from fleeing the redhat ecosystem, i am underwhelmed. i don't think that redhat really realizes yet how cheesed off this made a lot of people.

being shrewish about money is one thing; we can deal with that (see: oracle, IBM). it's really the complete unpredictability and how arrogantly hamfisted this centos debacle was. I can put up with a lot of shit in my job, but what i especially don't like or need is more unreliable unpredictable shit.

Oh, I think they know at this point.

https://devops.com/centos-rebellion-against-red-hat-gains-mo...

https://www.infoworld.com/article/3601202/red-hats-crime-aga...

https://arstechnica.com/gadgets/2020/12/centos-linux-is-gone...

Scientific Linux (sponsored by Fermilab) is a CentOS-like rebuild of RedHat Enterprise Linux and might also be worth a look. https://scientificlinux.org/

Fermilab discontinued work on Scientific Linux in 2019 in favor of using CentOS: https://listserv.fnal.gov/scripts/wa.exe?A2=SCIENTIFIC-LINUX...

It’s possible they could choose to restart work on Scientific Linux, but nothing has been announced yet.

To add, Fermi and Cern released a join statement

>CERN and Fermilab acknowledge the recent decision to shift focus from CentOS Linux to CentOS Stream, and the sudden change of the end of life of the CentOS 8 release. This may entail significant consequences for the worldwide particle physics community. We are currently investigating together the best path forward. We will keep you informed about any developments in this area during Q1 2021.

https://news.fnal.gov/2020/12/joint-cern-fermilab-statement-...

I suspect this is a reaction, not something planned a while back.

Yeah but they should expect reactions.

How did redhat kill centos again?

Reply on news.ycombinator.com