Comment by op00to | Hacker News Reader

Comment by op00to | Hacker News Reader

What a weird way to spend federal money. I say it's weird because NASA almost certainly has its own internal Linux support paid through overhead in one form or another, and again the incremental cost of adding "developer" support from Red hat is way, way cheaper than Rocky.

Finally, I doubt Rocky's support can perform as well as Red Hat's. No, I'm not talking about the people who talk on the phone when you break something. If you find a bug in a package, will Rocky be able to quickly and effectively upstream the fix, or will Rocky end up maintaining you on a custom patchlevel forever?

I doubt Rocky has the ability to truly fulfill 24/7 support. It's difficult to build a deep bench for support, and nearly impossible to make sure you're keeping up 24/7 capability for other Maintenance Engineering type tasks.

And if none of these reasons are important enough to stop you from paying them, maybe the thing you're doing isn't important enough to warrant paying for support anyway.

Former NASA/JPL employee. I can say with gusto that Linux was never really supported by overhead. Projects pay directly for cloud services, servers, computer time, equipment for field testing, etc. Those are where rocky Linux would be used. At least in my experience, the usual burden funds are used for vanilla windows desktops or laptops for employees, which works fine for 90% of the org. The rest write proposals to get non standard equipment in one form or another. This drastically reduces overhead costs and allows results driven grants to buy only necessary equipment. Same as any other gov funded research-related capital purchase. At least in theory.

To be honest, you sound more like a Red Hat fanboi than someone who is making persuasive arguments.

For one, you're implying that Red Hat can fix bugs and can upstream them more quickly than Rocky. Have you seen the amount of time it takes Red Hat to move fixes in to sources? That alone is silly to even mention. I don't know how long Rocky would take, but in the entire world, hardly any companies take as long as Red Hat.

Two, do you think Red Hat "truly fulfill(s) 24/7 support"? Sure, you can get someone on the phone, but how many hours will it take before you can get someone who knows anything? All that navigation and escalation is time consuming. I'd rather wait for an expert who knows the area of my problem intimately to be woken up to contact me after an hour than be appeased by being on the phone for three hours explaining a problem to person after person who won't directly escalate and wants to hear the problem explained.

"maybe the thing you're doing isn't important enough to warrant paying for support anyway." Huh? That's just ridiculous.

To be honest, you sound more like recent college graduate with minimal (no) experience working in enterprise.

You probably don't realize this given your lack of experience, but when you are running FOSS powered/adjacent enterprise equipment (think storage systems and compute fabrics) the bugs you encounter are often not things that simply show up on the first page of google.

There often isn't anyone to "wait" for (other than the resources you pay for), which is why if you value uptime and availability, you PAY for a support contract.

I was a staff level systems engineer at a major movie studio in California (owned a 24+PB tape library and a few petabytes of fast storage), and I can confidently say all of our dedicated support assets were valuable members of our broader team (first name basis, ask questions/raise issues with a text message, etc...).

In the rare occasion an engineer wasn't cutting it we would have our TAM get us a new one. Typically you are literally paying these people's salary as part of your contract so these are decisions that you as a customer actually have a lot of input into.

If you'd rather wait for someone to answer you on StackOverflow that's a decision you can make, but if you pull that move in any serious enterprise you will terminated pretty quickly.

The whole point of Rocky is to be bug-for-bug compatible with RHEL; your opinion on who can fix bugs faster is completely orthogonal to this discussion.

I'm flattered that I sound like a recent college graduate :) I've been a Unix systems administrator for more than a quarter of a century.

Wild generalization: Red Hat is mostly bullshit. Their support is good at handling the myriad quirks and bull they've intentionally added to their OS to differentiate their product, but they're not so good at things beyond that, in my experience. I'm generalizing, of course, but in too many instances Red Hat has been unique in having issues that other common OSes don't. In many environments I've set up proper servers with other OSes for comparison so I'd know when something is a Red Hat-ism.

If you're somehow authoritative because you've worked for a major movie studio with petabytes of storage, then I suppose I'm authoritative, too. I have and still work for major movie studios and I handle petabytes of storage, too :) Perhaps we know each other.

Red Hat couldn't get their OS to see all the zones of a StorNext, while at the very same moment identical hardware next to that Red Hat system running SuSE worked fine. Drives were swapped between the systems, and the problem followed Red Hat. It was a clean install. Red Hat support told us they couldn't do anything and said it was a StorNext problem. We contacted StorNext, and they were even willing to work with Red Hat. Red Hat required us to have a three way phone call to include the StorNext people and wouldn't answer their emails. I had to create email accounts for the StorNext people. It was extremely unprofessional and nothing moved on this until the company threatened to cancel Red Hat support entirely.

Unfortunately, my other experiences were more reminiscent of dealing with AT&T than dealing with an organization that wants to make things work.

> Have you seen the amount of time it takes Red Hat to move fixes in to sources?

They will absolutely cut you a patch to use on your systems. Upstreaming does take time, yes, but that doesn't mean the person experiencing the problem has to wait to get their issue resolved.

> I say it's weird because NASA almost certainly has its own internal Linux support paid through overhead in one form or another

Why do you presume this?

(Former NASA contractor here: they don't.)

Not who you are replying to but I would expect any large organization (eg: almost every university and large research institute) to have a competent IT team that can help with most problems.

I would not expect them to make patches to packages or the kernel and I do not know how IT works, but I suspect this is the reason they presumed what they presumed.

Honestly I suspect rocky support to be faster than red hat.

What makes you think a company with 89 employees that just clones Red Hat's source code would be faster than Red Hat with 19,000?

The mythical man month.

More people can very easily lead to nothing getting done.

This doesn't really apply to support. With support, there is generally 1 engineer assigned to one customer for the life of the case. If a patch is needed, a software maintenance engineer may be added. You can't throw extra people at a support case and expect it to complete faster. You can, however, throw more people at a support TEAM and handle more cases.

So Rocky needs to assign a single engineer to NASA support then. How does 19,000 people help with that?

Because surely they have other customers than these three workstations.

It's not 3 workstations, it's 3x per person support.

Point still stands, when you have an 80 man team you can’t have many customers who require your undivided attention.

That's typically what RedHat does...

If you're a big account you get your own Technical Account Manager (TAM) and a dedicated support engineer.

This is like bread and butter for anyone with any experience in enterprise.

How is HN so green in this regard?

No I'm saying if you're assigning *a person to a contract* Rocky can easily do that too.

Price's law, for one. Experience with the general effectiveness of large companies, especially their support teams, for another.

If Price's law is at play, then there will still be more competent people doing the work in Red Hat's support team than Rocky's.

How likely is it that there's someone on Rocky's staff of 80 who...

a. is an expert in the code, and has worked on it on customers behalf before

b. is able to effectively support a customer in addition to fixing a bug

c. is awake at the time you need them

d. isn't totally underwater with other issues

How likely is it that there's someone in Red Hat's staff of 20,000 (or whatever) who fit the same criteria?

You'll need experts in kernel, services, storage, networking, filesystems, virtualization / containers. Enterprise support shouldn't be done by generalists, it should be done by someone who has depth in the particular problem space.

Red Hat is supporting a lot more subscribers, I'd think? I have no experience with Red Hat phone support, their product has always been great and their online resources answered any question I had.

My impression from access.redhat.com is that much of the written material comes from support given to actual customers.

Red Hat (any many other companies) follows "knowledge centered support". [1] Essentially the knowledge base is the "workspace" or scratch paper of the support case, and as information comes in the case, the information should be organized in the knowledge base solution such that by the end, you've got the bones of a solid solution other customers can follow.

It's a whole Thing that People charge huge sums to implement, but the idea itself is pretty solid.

[1] https://www.serviceinnovation.org/kcs/

On multiple occasions, I've had Red Hat take 6 months to resolve a paid support case to fix something that already had a fix plus test cases committed upstream, and that could be cherry-picked into their version with no conflicts.

"Take 6 months" sounds a lot like "distribute the fix on the next minor update rather than doing so asynchronously".

Precisely because it has 89 employees instead of 19,000, it can be MUCH faster.

Have you ever tried to get support from Red Hat? Honestly? The first couple of tiers are often people who are barely computer literate. I've talked with several people who - I'm not kidding - had no idea what the Power architecture is and had no clue that there are architectures aside from x86.

Have you tried getting support from RedHat as an actual enterprise customer rather than an SMB?

Obviously if you pay for the lowest level of support, that's what you are going to get. Serious businesses pay for dedicated account resources. This typically includes a Technical Account Manager (TAM) and a dedicated support resource that works along side Professional Services to take care of your install.

When done _properly_ you will have nearly instant support from an excellent engineer.

Not sure who you were talking to but Red Hat support doesn't use tiers, cases go directly to SMEs on the feature and from there the only escalation point is the actual engineers working on the lines of code affected.

Did you watch the movie 300? Then you know

> No, I'm not talking about the people who talk on the phone when you break something. If you find a bug in a package, will Rocky be able to quickly and effectively upstream the fix, or will Rocky end up maintaining you on a custom patchlevel forever?

I'm intrigued. What makes you think Rocky will be faster than Red Hat? How does Rocky handle that situation? Do they have COPR repos or similar that you add to your system? What do they do if the patch gets rejected upstream?

Is your car dealership faster at changing the tires and oil on your car than a tire or oil-changing company would be?

RHEL is effectively a bunch of FOSS bundled and rebranded. Rocky is effectively RHEL with another paint job. If a patch gets rejected upstream, Red Hat is known to reject the author/maintainer's rationale for this rejection and, over years, may then extend, embrace, and extinguish that community, replacing and repainting it as their product.

Sometimes this is a community-serving change (i.e., docker -> (OCI+) -> podman), and sometimes it's a community-replacing change (i.e., Kubernetes -> Origin/OKD -> Openshift). In all cases, it's a redeclaration of who actually is the most knowledgeable expert. Management re-assigning experts is not necessarily as aligned as the merit of an original engineering team/community providing their solution.

I'd rather Rocky's solution than the walled-garden in a cathedral courtyard approach. If something cannot be pushed upstream (i.e., user-hostile defaults, vendor lock-in, or arbitrary rebranding), this blockage is sometimes a feature we all want for FOSS, even while it may not give privileges to those who can throw more money at the problem under the condition it will then be able to extract more profit.

The purpose of Rocky is to be bug-for-bug compatible with RHEL. What makes you think they would ever attempt to upstream something?

We do all the time.

They probably do it like Debian. "Here is a patch, looks good, just ship it."

With Red Hat it would have to get approved and tested first.

I suspect Rocky support is three people in Europe somewhere with maybe one or two others in North America. I also suspect Rocky has no Support Engineering group, so their own product engineering resources must be redirected every time a customer needs a patch.

If Rocky does have a Support Engineering group, they aren't regular committers to the kernel, and so most common bugs found will take much longer to be upstreamed. This increases the load on Engineering because they'll have to carry that fix until it makes it into upstream and the customer upgrades.

I have seen no indication that Rocky has a robust QA infrastructure for customer patches.

Note the Rocky support model allows each "person" to only have two cases open at a time. This indicates they're forced to limit capacity.

Let's say I'm wrong about all of this. Let's say Rocky support is better than Red Hat. I've seen what happens when support scales from a small company to a large one. If ever you got good support, those days are over if Rocky sees success and the support group scales.

It takes a lot to design and build a car and manage its dealerships. It takes orders of magnitude less effort for a mechanic to maintain it, and even less to simply paint and clean it. Red Hat builds this car - Rocky repaints it and acts far more like Jiffy Lube than like a dealership. Have you ever seen how long it takes, and how much more it costs, to have your oil changed by the dealership? Those focusing on the specialty are able to deliver on it more efficiently and with less expense to all involved.

I've seen what happens when you take your car to Jiffy Lube and the minimum wage oil tech they hire doesn't tighten the oil drain plug all the way.

Or strips it.

The support is provided by a company called CIQ rather than the engineers behind Rocky itself. You can even see their support tiers on their website (NASA picked Advanced)

https://ciq.com/support/rocky-linux/

The difference is with RH you can often get escalated up to the engineer writing the thing. I know I did when working for companies paying for that support. There's usually a few layers of rather useless L1 and L2 though.

Those lower layers are a significant anti-feature for a customer needing quick, effective support, and it is only occasionally/rarely the case that a key author or maintainer is actually employed by Red Hat.

Most things delivered by Red Hat were not written by Red Hat and where they are, that delivery tends to be a rewritten, renamed project which went from community-driven (FOSS made to serve the user) into a corporate-driven model (gate-kept software which serves primarily as a profitable subscription-dependant product).

The trade-off is effectively for legal liability transference more than for genuine supportability. I would rather get eventual real support from the bazaar of people who wrote a thing (knowing, intimately and technically why) than liability-waving support (assuming successful escalation) from those who rewrote it in the cathedral with divergence intentionally created for corporate control and profit.

Actually most of the important bits things delivered by Red Hat are written by Red Hat employees. Or at least maintained.

Reply on news.ycombinator.com