Hacker News Reader

Comment by aaomidi | Hacker News Reader

Honestly I suspect rocky support to be faster than red hat.

I suspect Rocky support is three people in Europe somewhere with maybe one or two others in North America. I also suspect Rocky has no Support Engineering group, so their own product engineering resources must be redirected every time a customer needs a patch.

If Rocky does have a Support Engineering group, they aren't regular committers to the kernel, and so most common bugs found will take much longer to be upstreamed. This increases the load on Engineering because they'll have to carry that fix until it makes it into upstream and the customer upgrades.

I have seen no indication that Rocky has a robust QA infrastructure for customer patches.

Note the Rocky support model allows each "person" to only have two cases open at a time. This indicates they're forced to limit capacity.

Let's say I'm wrong about all of this. Let's say Rocky support is better than Red Hat. I've seen what happens when support scales from a small company to a large one. If ever you got good support, those days are over if Rocky sees success and the support group scales.

It takes a lot to design and build a car and manage its dealerships. It takes orders of magnitude less effort for a mechanic to maintain it, and even less to simply paint and clean it. Red Hat builds this car - Rocky repaints it and acts far more like Jiffy Lube than like a dealership. Have you ever seen how long it takes, and how much more it costs, to have your oil changed by the dealership? Those focusing on the specialty are able to deliver on it more efficiently and with less expense to all involved.

I've seen what happens when you take your car to Jiffy Lube and the minimum wage oil tech they hire doesn't tighten the oil drain plug all the way.

Or strips it.

The support is provided by a company called CIQ rather than the engineers behind Rocky itself. You can even see their support tiers on their website (NASA picked Advanced)

https://ciq.com/support/rocky-linux/

The difference is with RH you can often get escalated up to the engineer writing the thing. I know I did when working for companies paying for that support. There's usually a few layers of rather useless L1 and L2 though.

Those lower layers are a significant anti-feature for a customer needing quick, effective support, and it is only occasionally/rarely the case that a key author or maintainer is actually employed by Red Hat.

Most things delivered by Red Hat were not written by Red Hat and where they are, that delivery tends to be a rewritten, renamed project which went from community-driven (FOSS made to serve the user) into a corporate-driven model (gate-kept software which serves primarily as a profitable subscription-dependant product).

The trade-off is effectively for legal liability transference more than for genuine supportability. I would rather get eventual real support from the bazaar of people who wrote a thing (knowing, intimately and technically why) than liability-waving support (assuming successful escalation) from those who rewrote it in the cathedral with divergence intentionally created for corporate control and profit.

Actually most of the important bits things delivered by Red Hat are written by Red Hat employees. Or at least maintained.

What makes you think a company with 89 employees that just clones Red Hat's source code would be faster than Red Hat with 19,000?

The mythical man month.

More people can very easily lead to nothing getting done.

This doesn't really apply to support. With support, there is generally 1 engineer assigned to one customer for the life of the case. If a patch is needed, a software maintenance engineer may be added. You can't throw extra people at a support case and expect it to complete faster. You can, however, throw more people at a support TEAM and handle more cases.

So Rocky needs to assign a single engineer to NASA support then. How does 19,000 people help with that?

Because surely they have other customers than these three workstations.

It's not 3 workstations, it's 3x per person support.

Point still stands, when you have an 80 man team you can’t have many customers who require your undivided attention.

That's typically what RedHat does...

If you're a big account you get your own Technical Account Manager (TAM) and a dedicated support engineer.

This is like bread and butter for anyone with any experience in enterprise.

How is HN so green in this regard?

No I'm saying if you're assigning *a person to a contract* Rocky can easily do that too.

Price's law, for one. Experience with the general effectiveness of large companies, especially their support teams, for another.

If Price's law is at play, then there will still be more competent people doing the work in Red Hat's support team than Rocky's.

How likely is it that there's someone on Rocky's staff of 80 who...

a. is an expert in the code, and has worked on it on customers behalf before

b. is able to effectively support a customer in addition to fixing a bug

c. is awake at the time you need them

d. isn't totally underwater with other issues

How likely is it that there's someone in Red Hat's staff of 20,000 (or whatever) who fit the same criteria?

You'll need experts in kernel, services, storage, networking, filesystems, virtualization / containers. Enterprise support shouldn't be done by generalists, it should be done by someone who has depth in the particular problem space.

Red Hat is supporting a lot more subscribers, I'd think? I have no experience with Red Hat phone support, their product has always been great and their online resources answered any question I had.

My impression from access.redhat.com is that much of the written material comes from support given to actual customers.

Red Hat (any many other companies) follows "knowledge centered support". [1] Essentially the knowledge base is the "workspace" or scratch paper of the support case, and as information comes in the case, the information should be organized in the knowledge base solution such that by the end, you've got the bones of a solid solution other customers can follow.

It's a whole Thing that People charge huge sums to implement, but the idea itself is pretty solid.

[1] https://www.serviceinnovation.org/kcs/

On multiple occasions, I've had Red Hat take 6 months to resolve a paid support case to fix something that already had a fix plus test cases committed upstream, and that could be cherry-picked into their version with no conflicts.

"Take 6 months" sounds a lot like "distribute the fix on the next minor update rather than doing so asynchronously".

Precisely because it has 89 employees instead of 19,000, it can be MUCH faster.

Have you ever tried to get support from Red Hat? Honestly? The first couple of tiers are often people who are barely computer literate. I've talked with several people who - I'm not kidding - had no idea what the Power architecture is and had no clue that there are architectures aside from x86.

Have you tried getting support from RedHat as an actual enterprise customer rather than an SMB?

Obviously if you pay for the lowest level of support, that's what you are going to get. Serious businesses pay for dedicated account resources. This typically includes a Technical Account Manager (TAM) and a dedicated support resource that works along side Professional Services to take care of your install.

When done _properly_ you will have nearly instant support from an excellent engineer.

Not sure who you were talking to but Red Hat support doesn't use tiers, cases go directly to SMEs on the feature and from there the only escalation point is the actual engineers working on the lines of code affected.

Did you watch the movie 300? Then you know

> No, I'm not talking about the people who talk on the phone when you break something. If you find a bug in a package, will Rocky be able to quickly and effectively upstream the fix, or will Rocky end up maintaining you on a custom patchlevel forever?

I'm intrigued. What makes you think Rocky will be faster than Red Hat? How does Rocky handle that situation? Do they have COPR repos or similar that you add to your system? What do they do if the patch gets rejected upstream?

Is your car dealership faster at changing the tires and oil on your car than a tire or oil-changing company would be?

RHEL is effectively a bunch of FOSS bundled and rebranded. Rocky is effectively RHEL with another paint job. If a patch gets rejected upstream, Red Hat is known to reject the author/maintainer's rationale for this rejection and, over years, may then extend, embrace, and extinguish that community, replacing and repainting it as their product.

Sometimes this is a community-serving change (i.e., docker -> (OCI+) -> podman), and sometimes it's a community-replacing change (i.e., Kubernetes -> Origin/OKD -> Openshift). In all cases, it's a redeclaration of who actually is the most knowledgeable expert. Management re-assigning experts is not necessarily as aligned as the merit of an original engineering team/community providing their solution.

I'd rather Rocky's solution than the walled-garden in a cathedral courtyard approach. If something cannot be pushed upstream (i.e., user-hostile defaults, vendor lock-in, or arbitrary rebranding), this blockage is sometimes a feature we all want for FOSS, even while it may not give privileges to those who can throw more money at the problem under the condition it will then be able to extract more profit.

The purpose of Rocky is to be bug-for-bug compatible with RHEL. What makes you think they would ever attempt to upstream something?

We do all the time.

They probably do it like Debian. "Here is a patch, looks good, just ship it."

With Red Hat it would have to get approved and tested first.

Reply on news.ycombinator.com