NASA secures contract with Rocky Enterprise Linux
sam.gov
sam.gov
To me, it's more like 3 engineers at NASA are interested in it and got their boss to get them 3 licenses? As opposed to "NASA moves all RHEL licenses to Rocky after Redhat destroys CentOS community"?
Edit: I know nothing of how govournment agencies (and certainly not US ones) work.. I assume all purchases go through some process where they're all made public?
"The item described in the table below is for the FY23 CIQ Rocky Enterprise Linux Renewal."
Since I retired on Friday, my experience with government IT and procurement is slightly out of date but based upon my knowledge of it until that point I would consider it a better wild ass guess than NASA procured 3 Rocky Linux workstation support contracts to test it before making some agency wide switch.
Thanks, I almost spit my coffee.
Some years ago I read some docs on how exactly RHEL licensing works... guess the team which Oraclised MSFT in 2010 moved to RH.
* The institution has a slush / IT fund that provides basic windows laptops or MACs, right off the shelf with lots of security add-ons
* For any non standard equipment or software, the engineers on a project must seek money from the project's budget and justify that cost to the project manager. (not their long-term boss).
* Project managers have a fixed budget based on the proposal they wrote to the funding agency, usually US GOV, with a fixed cost for these kinds of purchases, agreed upon at the time the proposal was funded
* The limited project funds are used to purchase whatever is needed to satisfy the project's objectives. If the project's objectives are not satisfied, the funding agency can stop, or even pull back funding from the project manager
* The approvals for equipment purchases are, at least in theory, given based on merit, which is very often weighed using prior successful proposals. New types of purchases are scrutinized more. If it's traditional to buy CENTOS, then buying ROCKY would be more difficult.
* That precedent is key. Once it becomes clear that ROCKY is being used by successful teams, then the approval process is much easier.
That's why this is a big-ish deal (not huge, but nice to see). Now, people writing proposals and working on projects can point to this as precedent. The best outcome is when funding agencies see "ROCKY is better and more cost effective for the teams we've been funding", then you might get a funder to request that teams use ROCKY as part of the proposal requirements. This is almost nonexistent in federal contracts, but is quite common when dealing with military or commercial contracts.
That, and ROCKY can probably now have a NASA logo on their website under "customers", which adds a bunch of credibility to the project, honestly, whether it's necessarily deserved or not based on 3 desktops.
Purchases were hard because acquisition was hard but I don’t think the bar requires previous use. Usually getting a vendor to accept a PO was the hard part IIRC
I sort-of assumed that for big purchases it would be a big procedure, as you've stated.
I had it in my mind, there'd be like a lower threshold (i.e. licenses for one or two things), which would go under the radar and just be purchased when someone requested it (or end of year "budget needs to be spend" - you know, that sort of old school "budget per department - spend it or you lose it" sort of thing.) :D
This also went with the idea that the "3 x CIQ Rocky Enterprise Linux Per Person Advanced - Annual Subscription Service Period" would be a small purchase - though of course, I don't _really_ have any idea what this means - could be a massive 24x7 support with 3 Rocky Linux support staff on-call - who knows :)
> That, and ROCKY can probably now have a NASA logo on their website under "customers", which adds a bunch of credibility to the project, honestly, whether it's necessarily deserved or not based on 3 desktops.
This on the other hand is verry interesting!! Thanks for pointing this out :)
Edit: sorry, I misread your comment :( Thought you said their logo _was_ on the Rocky site :P
I'm not so convinced based on my experience. Seems like that could be the ideal, but in practice for such a large institution it seems difficult to implement. Sharing the word that this team over here is using XYZ and seeing good results to that team over there is challenging.
Link submitted by someone at https://news.ycombinator.com/item?id=36417070
So, RedHat can satisfy the letter of the law regarding licensing, where source is distributed under GPL to those who received the binaries. With each incremental binary distribution they satisfy the GPL rules and makes source available to the recipient of binaries, but they may cut off a customer as soon as they detect that the customer is actually exercising their full GPL rights. So that customer loses access to future binaries and corresponding sources.
This attitude violates the experience many of us take for granted about free software and the GPL, where we expect a community to be engaging as an equivalence class of members sharing the same rights. But the GPL started with provisions enabling this narrower sense. The source distribution could be offered individually to each recipient of binaries _or_ made generally available to the public independent of any existing exchange of binaries between two parties. So the GPL allowed the recipient to transitively distribute the sources they have received, but does not actually obligate the original distributor to also provide sources to other unrelated parties.
It seems important to note that the Debian Free Software Guidelines as applied by the Debian project specifically exclude any software that requires providing source to unrelated people this way (the “desert island” and “dissident” tests[1]). In practice, the “unrelated” people in license terms of that kind are usually the original copyright holders.
This kind of GPL not-a-workaround always made me feel a bit queasy, but open access to RH code (even if a bit stale and hard as hell to get to build) alleviated that feeling somewhat. I guess it no longer does.
RedHat has been giving them away to everyone; but they are not legally obligated to do so
To add to this, Red Hat does offers a free developer tier subscription, which allows individuals to download Enterprise Linux free of costs, but with strings attached. One of those strings, somewhere in the fine print is legal language about the restrictions of redistributing the software at the free tier. I'm not clear on the exact details, so I won't elaborate & please don't quote me, but it's my understanding that it's somehow a violation of terms redistribute. I'm not sure how that works with the GPL, and I'm guessing the bits ARE still shareable under the GPL, but the separate contract with Red Hat makes that a breach of contract. The mind boggles.
For those of us interested in investigating, could you provide any information to support these claims? It looks like FUD when these claims have no backing information associated with them -- but if there is more information available I would love to know.
According to his Linkedin[1] and an interview[2] the CEO of CloudLinux -- Igor Seletskiy -- lives in California. From some brief reading it seems his company is remote and I think I saw some content that he may have Russian employees, but didn't see anything notable that was worth linking here.
[0] https://en.m.wikipedia.org/wiki/CloudLinux_OS
[1] https://www.linkedin.com/in/iseletsk
[2] https://lowendbox.com/blog/interview-with-igor-seletskiy-ceo...
There's hardly any job openings now but there used to be a lot that required or strongly recommended Russian language. A couple of years ago, I was on the verge of working for TuxCare.
CloudLinux is a large financial sponsor of AlmaLinux still, and probably always will be. The rest of what you said couldn't be farther from the truth.
I wonder of Rocky got some real $ from NASA ?
Finally, I doubt Rocky's support can perform as well as Red Hat's. No, I'm not talking about the people who talk on the phone when you break something. If you find a bug in a package, will Rocky be able to quickly and effectively upstream the fix, or will Rocky end up maintaining you on a custom patchlevel forever?
I doubt Rocky has the ability to truly fulfill 24/7 support. It's difficult to build a deep bench for support, and nearly impossible to make sure you're keeping up 24/7 capability for other Maintenance Engineering type tasks.
And if none of these reasons are important enough to stop you from paying them, maybe the thing you're doing isn't important enough to warrant paying for support anyway.
More people can very easily lead to nothing getting done.
If you're a big account you get your own Technical Account Manager (TAM) and a dedicated support engineer.
This is like bread and butter for anyone with any experience in enterprise.
How is HN so green in this regard?
How likely is it that there's someone on Rocky's staff of 80 who...
a. is an expert in the code, and has worked on it on customers behalf before
b. is able to effectively support a customer in addition to fixing a bug
c. is awake at the time you need them
d. isn't totally underwater with other issues
How likely is it that there's someone in Red Hat's staff of 20,000 (or whatever) who fit the same criteria?
You'll need experts in kernel, services, storage, networking, filesystems, virtualization / containers. Enterprise support shouldn't be done by generalists, it should be done by someone who has depth in the particular problem space.
My impression from access.redhat.com is that much of the written material comes from support given to actual customers.
It's a whole Thing that People charge huge sums to implement, but the idea itself is pretty solid.
Have you ever tried to get support from Red Hat? Honestly? The first couple of tiers are often people who are barely computer literate. I've talked with several people who - I'm not kidding - had no idea what the Power architecture is and had no clue that there are architectures aside from x86.
Obviously if you pay for the lowest level of support, that's what you are going to get. Serious businesses pay for dedicated account resources. This typically includes a Technical Account Manager (TAM) and a dedicated support resource that works along side Professional Services to take care of your install.
When done _properly_ you will have nearly instant support from an excellent engineer.
I'm intrigued. What makes you think Rocky will be faster than Red Hat? How does Rocky handle that situation? Do they have COPR repos or similar that you add to your system? What do they do if the patch gets rejected upstream?
RHEL is effectively a bunch of FOSS bundled and rebranded. Rocky is effectively RHEL with another paint job. If a patch gets rejected upstream, Red Hat is known to reject the author/maintainer's rationale for this rejection and, over years, may then extend, embrace, and extinguish that community, replacing and repainting it as their product.
Sometimes this is a community-serving change (i.e., docker -> (OCI+) -> podman), and sometimes it's a community-replacing change (i.e., Kubernetes -> Origin/OKD -> Openshift). In all cases, it's a redeclaration of who actually is the most knowledgeable expert. Management re-assigning experts is not necessarily as aligned as the merit of an original engineering team/community providing their solution.
I'd rather Rocky's solution than the walled-garden in a cathedral courtyard approach. If something cannot be pushed upstream (i.e., user-hostile defaults, vendor lock-in, or arbitrary rebranding), this blockage is sometimes a feature we all want for FOSS, even while it may not give privileges to those who can throw more money at the problem under the condition it will then be able to extract more profit.
With Red Hat it would have to get approved and tested first.
If Rocky does have a Support Engineering group, they aren't regular committers to the kernel, and so most common bugs found will take much longer to be upstreamed. This increases the load on Engineering because they'll have to carry that fix until it makes it into upstream and the customer upgrades.
I have seen no indication that Rocky has a robust QA infrastructure for customer patches.
Note the Rocky support model allows each "person" to only have two cases open at a time. This indicates they're forced to limit capacity.
Let's say I'm wrong about all of this. Let's say Rocky support is better than Red Hat. I've seen what happens when support scales from a small company to a large one. If ever you got good support, those days are over if Rocky sees success and the support group scales.
Most things delivered by Red Hat were not written by Red Hat and where they are, that delivery tends to be a rewritten, renamed project which went from community-driven (FOSS made to serve the user) into a corporate-driven model (gate-kept software which serves primarily as a profitable subscription-dependant product).
The trade-off is effectively for legal liability transference more than for genuine supportability. I would rather get eventual real support from the bazaar of people who wrote a thing (knowing, intimately and technically why) than liability-waving support (assuming successful escalation) from those who rewrote it in the cathedral with divergence intentionally created for corporate control and profit.
For one, you're implying that Red Hat can fix bugs and can upstream them more quickly than Rocky. Have you seen the amount of time it takes Red Hat to move fixes in to sources? That alone is silly to even mention. I don't know how long Rocky would take, but in the entire world, hardly any companies take as long as Red Hat.
Two, do you think Red Hat "truly fulfill(s) 24/7 support"? Sure, you can get someone on the phone, but how many hours will it take before you can get someone who knows anything? All that navigation and escalation is time consuming. I'd rather wait for an expert who knows the area of my problem intimately to be woken up to contact me after an hour than be appeased by being on the phone for three hours explaining a problem to person after person who won't directly escalate and wants to hear the problem explained.
"maybe the thing you're doing isn't important enough to warrant paying for support anyway." Huh? That's just ridiculous.
You probably don't realize this given your lack of experience, but when you are running FOSS powered/adjacent enterprise equipment (think storage systems and compute fabrics) the bugs you encounter are often not things that simply show up on the first page of google.
There often isn't anyone to "wait" for (other than the resources you pay for), which is why if you value uptime and availability, you PAY for a support contract.
I was a staff level systems engineer at a major movie studio in California (owned a 24+PB tape library and a few petabytes of fast storage), and I can confidently say all of our dedicated support assets were valuable members of our broader team (first name basis, ask questions/raise issues with a text message, etc...).
In the rare occasion an engineer wasn't cutting it we would have our TAM get us a new one. Typically you are literally paying these people's salary as part of your contract so these are decisions that you as a customer actually have a lot of input into.
If you'd rather wait for someone to answer you on StackOverflow that's a decision you can make, but if you pull that move in any serious enterprise you will terminated pretty quickly.
The whole point of Rocky is to be bug-for-bug compatible with RHEL; your opinion on who can fix bugs faster is completely orthogonal to this discussion.
Wild generalization: Red Hat is mostly bullshit. Their support is good at handling the myriad quirks and bull they've intentionally added to their OS to differentiate their product, but they're not so good at things beyond that, in my experience. I'm generalizing, of course, but in too many instances Red Hat has been unique in having issues that other common OSes don't. In many environments I've set up proper servers with other OSes for comparison so I'd know when something is a Red Hat-ism.
If you're somehow authoritative because you've worked for a major movie studio with petabytes of storage, then I suppose I'm authoritative, too. I have and still work for major movie studios and I handle petabytes of storage, too :) Perhaps we know each other.
Red Hat couldn't get their OS to see all the zones of a StorNext, while at the very same moment identical hardware next to that Red Hat system running SuSE worked fine. Drives were swapped between the systems, and the problem followed Red Hat. It was a clean install. Red Hat support told us they couldn't do anything and said it was a StorNext problem. We contacted StorNext, and they were even willing to work with Red Hat. Red Hat required us to have a three way phone call to include the StorNext people and wouldn't answer their emails. I had to create email accounts for the StorNext people. It was extremely unprofessional and nothing moved on this until the company threatened to cancel Red Hat support entirely.
Unfortunately, my other experiences were more reminiscent of dealing with AT&T than dealing with an organization that wants to make things work.
They will absolutely cut you a patch to use on your systems. Upstreaming does take time, yes, but that doesn't mean the person experiencing the problem has to wait to get their issue resolved.
Why do you presume this?
(Former NASA contractor here: they don't.)
I would not expect them to make patches to packages or the kernel and I do not know how IT works, but I suspect this is the reason they presumed what they presumed.
If I was choosing a server OS today, I personally would probably pick Debian.
For these shops, you don't need the paid enterprise offering across the board (dev, testing, etc) so that is where an EL rebuild is handy. In those cases you don't want too much different between prod and test/dev, but you don't need the paid support or priority patches on internal test hosts.
RH would not be of much value if the support staff were not any more effective than the average end user.
Is this an ego thing?
It's only the boss that wants to pay for a safety net, forcing the issue of raising a ticket is their validation of money well spent I highly suspect. We're all human beings who are quite technical on this forum, you work for RH, the other guy works for someone else.
I have a suspicion that RH support is what the boss pays for as a backup should their employees be unable to solve a problem or decide to go to pastures new, as people do from time to time.
they're not, because they have no idea what they're looking for, they lack the subject knowledge and expertise, and they werent involved with directly producing a lot of the open source components they are having problems with.
By "engineer" I mean, "we are the people that wrote the actual open source components they are having a problem with". It would not be in their interests to hire us directly because 95% of the time the regular support people can do everything they need without things escalated to engineering.
My gripe about the original thread was bosses interrupting that investigation and telling the onsite staff to open a case elsewhere, IMO, that's disruptive behaviour.
I dont think this really happens, when we get these customer issues and the people who have tried to fix the problem are on the phone, they are lost / panicked / at a dead end. They are not like, "yeah we think it's ABC but our boss told us to call you". that's not really a thing. if customers can fix the problem, they do. they are not in a hurry to call redhat for things they can do themselves and they're not really supposed to since their support minutes are a finite resource.
What typically happens in this situation is the chap logging the ticket gets pissed off and says to the boss, "ticket logged it's with $vendor, I guess it doesn't need babysitting so I'm working on other tickets now".
The boss in this case is happy as it shows their earlier investment was a fine idea. The employee though is disgruntled as their toys were taken away and things it's not such a fine place to work.
For just ease of management and customization can't beat vanilla Debian
If you run the standard installer from a vanilla USB image, you will be asked questions about the config, most of which produce reasonable results if you just hit <enter>. Some of them will fail in certain environments. (No DHCP server? You will need to enter IP networking details.)
If you want to install a fleet of Debian machines (or VM images), you set up a PXE server and a set of answers to the installation questions, or you build an image for direct deployment, or you use the Debian VM image, or... whatever: there are a lot of possibilities that will work.
Most of this has been true for at least 3 Stables.
For home server use: Tumbleweed, Leap or MicroOS depending on your personality. :)
RHEL and its clones are supported with updates for 10 years, often ship with "more modern" server software than Debian does (e.g. firewalld, podman, systemd etc) and DNF, whilst slower, still blows apt in terms of feature set (e.g. transcriptional, roll-backs, history, offline upgrades)
There's also SUSE.
> There's also SUSE.
So, why doesn't SUSE get close?
My understanding is that the problem with RHEL clones is just that, that they are clones.
From what I've read, the problem is that RHEL will often make changes to suit their various enterprise customers, for example I seem to recall reading somewhere that OpenSSL was one of many things "touched" in that way.
I hear many good things about DNF vs apt, but I'm too worried by the skeletons in the clone cupboard.
It's terrifying actually.
That's not really true. They often ship with Red Hat specific software (like Podman is) replacing widely used software (e.g. Docker, to which Podman still isn't feature complete even though it has been shoved in everyone's throats for a couple of years now), but "more modern" will vary wildly based on the release date.
Sooo much less fuckery, packages are just there and don't require installing/buying some random addon just to have DRBD+pacemaker cluster (that RHEL for some reason put into separate repo and subscription).
Ubuntu LTS is long enough for me, and seems to work without complaints.
At home I run Gentoo because I like to fomit and funroll.
For personal machines, I don't mind Fedora Server (or CentOS Stream) because I can handle the more frequent upgrade cycle.
Major version upgrades have less risk of breaking, while rollbacks become an option. And the attack surface is lower.
If unstable then Centos or Fedora.
I also have to use Ubuntu and Debian, and they are miles below Redhat derivates.
Day to day they should be by design, interchangeable. Rocky has a nicer logo... This kind of ugly exchange on reddit[0] always sort of soured my impression of Rocky though, didn't really pass the lebowski test.
0. https://www.reddit.com/r/linux/comments/qv6mg2/were_the_alma...
9.2 Release for PowerPC (ppc64le) architecture held back
During testing, we discovered an architecture-specific issue on ppc64le systems with the bundled version of Python 3.9. This issue not only prevents installing, but may break existing installations.
This issue is reproducible on CentOS Stream 9 and RHEL 9.2. We have opened a bug report upstream at rhbz#2203919 and are working to fix the issue.
Seems like they hold back releases to do additional testing. In this case, they avoided a bug that was present on RHEL (and presumably AlmaLinux).
I think many IT employees have a vested interest in Red Hat and the direction of "enterprise Linux". Seeing an organization like NASA show a willingness to use a Red Hat alternative gives credibility to that alternative.
So let's say that some Rocky or Alma Linux user or developer wants to make a change. They are RHEL clones, so they can't do that. But they can now contribute the change to CentOS Stream, which is the upstream for both RHEL and their own distro. That change can be reviewed and tested by the community + Red Hat, and then land in RHEL a few months later.
This isn't theoretical, to use one example, Facebook uses CentOS Stream in production and contributes back a significant amount.
Disclosure: I do work for Red Hat. But it never made sense to me why people say that the "CentOS community was destroyed" when it didn't really have much of a community, it had _users_, which is different. It has more of a "community" now than it did previously, because the development is actually open as opposed to Android-style "throwing bundles of code over the wall".
Pretty sure it's just for free beta testing
NASA seems to be taking the lead here and saying "we're going with Rocky". I think this move will encourage others on the fence to do the same, and build more of a community around the product not owned/maintained by Amazon.
CentOS Stream is not "bleeding edge". Every update that goes into CentOS Stream has already gone through the Red Hat QA process, and the updates are restricted to only the kinds of things which previously would have gone into RHEL. That is, mostly security updates and bugfixes.
It's a little bit ahead of RHEL, but it's not like Fedora, much less Rawhide.
Given the stage of the lifecycle, you'd be pretty silly to adopt AL2 on-prem at this point. AL2023 probably won't be appropriate either. Rocky is the way to go.
CentOS is now bleeding edge? Would you describe Debian Stable as bleeding edge also? On your contiunuum, where would you place Arch and Fedora?
> Rocky Linux is a Linux distribution developed by Rocky Enterprise Software Foundation, which is a privately owned benefit corporation that describes itself as a "self-imposed not-for-profit".
Is this only a structure set up to fund whatever needed to be done to rebrand RHEL? Or does anyone have shares that could be worth significant money?
Federal agencies can't really use CentOS after 2024-06-30, so everyone dealing with federal clients is working on an upgrade path (CentOS was a major solution due to piggy backing on RHEL certifications )
You have to either go with another distro or provide your own support (essentially your own fork).
Those who want to stay with RHEL-like but not deal with actual RHEL licensing are often going with Alma or Rocky.
What if they surplant RedHat, can they afford to undermine RedHat long term, or is there a point where they fork?
Rocky Linux, and CentOS before it, is a strange product for people who want the benefits of RHEL, but don't want to pay for RedHat work. If you don't want support, then I think is fine, that the cost RedHat pays to be an open source company, but when you then turn to another company who rebadge RHEL and charge for support, it gets a little weird.
The problem will only manifest itself if RedHats business is sufficiently underminded. Losing a few customers isn't an issue, and many of the users of Rocky Linux probably wasn't going to buy a RHEL licens anyway.
From the consumer side, if you buy a dropshipped product you don't get the benefits of working with the manufacturer, and are risking worse customer service.