Red Hat's response to the "obfuscated" source code accusations | Hacker News Reader

Red Hat's response to the "obfuscated" source code accusations

press.redhat.com

71 pointsTomasSedovic75 comments

Here's the meat since 80% of the article is PR blabbering and we all already know who Red Hat is:

When we released RHEL 6 approximately four months ago, we changed the release of the kernel package to have all our patches pre-applied. Why did we make this change? To speak bluntly, the competitive landscape has changed. Our competitors in the Enterprise Linux market have changed their commercial approach from building and competing on their own customized Linux distributions, to one where they directly approach our customers offering to support RHEL.

Frankly, our response is to compete. Essential knowledge that our customers have relied on to support their RHEL environments will increasingly only be available under subscription. The itemization of kernel patches that correlate with articles in our knowledge base is no longer available to our competitors, but rather only to our customers who have recognized the value of RHEL and have thus indirectly funded Red Hat’s contributions to open source that will advance their business now and in the future.

TL;DR: Competition is hard, especially when we are "open" and everyone else is not, so we adopted our competitors' tactics.

"we adopted our competitors' tactics."

No, Red Hat develops software, Oracle mooches off Red Hat's work. They have made this change to ensure that they can stay in business, which is important because unlike Oracle, they actually do work on the software they ship.

They can't adopt Oracle's tactics because there is nobody else doing what Red Hat does.

They can't adopt Oracle's tactics because there is nobody else doing what Red Hat does.

SUSE does what Red Hat does (work on the software it ships). In addition to that it also provides Enterprise Support for RHEL :). Which might make it both good and evil you could say or maybe just chameleonic.

Not to anywhere even close to approaching Red Hat's extend, and RHEL certainly isn't just repackaging SLES so my point remains.

It is true, however this is a bullshit example of loopholes of open source software. However to RH's defense I do see why they would do this. However part 2: RH is still mooching off the open source work of worldwide contributors. They need to respect that.

> Not to anywhere even close to approaching Red Hat's extend

This needs a citation.

I for one do not agree with you. Looking at the statistics of top kernel contributors for kernel 2.6.35 by lines changed¹, Red Hat can claim 7.8% of lines changed while Novell (SuSE) is at 4.7%. The difference is considerable but I do not think it's fair to say that that it's "Not to anywhere even close to approaching Red Hat's extend[sic]".

These statistics obviously cover just the kernel. The Linux environment is much larger than that, and it should be pointed out that SuSE is _the_ major contributor to KDE with openSUSE being generally regarded as the top-of-the-line KDE distribution. Novell also contributes significantly to GNOME.

I do not want to turn this comment into a laundry list of SuSE contributions to Linux, so I'll just say that IMO SuSE are certainly holding their own as the 2nd largest enterprise Linux vendor in terms of contribution to the general Linux ecosystem. If you do not agree, I would appreciate it if you could back your opinion with substantial evidence as opposed to just making unsubstantiated claims.

¹ http://lwn.net/Articles/395961/

I was under the impression that Novell no longer works on Gnome much, and that the torch (and many of the engineers) have moved on to Red Hat and some smaller companies (Igalia, etc).

"These statistics obviously cover just the kernel. The Linux environment is much larger than that..."

Being strict, actually isn't Linux just a kernel? Just being a BSD troll.

They became more closed (patch information is no longer available to anyone but customers). So they adopted one tactic, closenesses, of one competitor, Oracle.

Obviously, they didn't adopt all of Oracle's tactics. Nobody else is doing what Oracle does either (though we can be happy about that).

So what is the thought on Btrfs? Is Oracle not contributing back with that project?

I am no Oracle fan but I'm already using Btrfs and am pleased to finally have something in the same vein as ZFS for Linux.

I was reading that and at the pre-last paragraph just thought "has my brain just gotten hit in the balls?"

"We're committed to open source ... open source makes the best code in the world ... were innovators in open source ... we just decided to give ourselves a competitive advantage" ... WTF!?

Its like in one breath he snuck it in. Bullcrap.

How exactly is RedHat not showing a commitment to open source? Do they invest in open source? CHECK. Do they contribute to the adoption of open source? CHECK. Are they abiding by the open source licenses that govern the use of the open source on which they rely? CHECK.

All they have done in this case is make a legal, wise, and likely necessary move to protect themselves. Nothing they have done damages the kernel or prevents you from getting it. Nothing they have done damages any other OS projects.

Sounds like you've got a whole mouthful of sour grapes there...

They're withholding documentation for "open source" code they wrote. You're welcome to argue that's okay, sound business, etc. but if they were committed to open source they would release documentation alongside their code.

And until recently, they did.

They are withholding the intermediate steps between their kernel and a vanilla one. They are not withholding the source code - you can study it, change it and even diff it against the vanilla kernel to see what they changed.

I bet some comments may even help you figure out why they made a specific change.

Unfortunately, I believe all Oracle will just pay some proxy for RH support and get the same individual patches.

It sounds like the problem you both have is a high sense of entitlement for something that the license doesn't entitle you to have...why is Red Hat at fault for asking you to pay for something that wasn't at all free or easy to create?

Where did I say they were at fault? All I said was that they're making a significant move away from their open source business model.

Now in your post that I was replying to, on the other hand, you clearly seem to be talking about the broader social contract that the open source community is organized around. Red Hat makes the best code and documentation they have available to anyone, and in return the community does the same.

Red Hat has loudly trumpeted the superiority of this social contract over the proprietary avenue. Now their documentation has become proprietary. That says to me they believe the social contract and the business model they've built around it are somehow flawed.

Personally, I don't have a dog in this fight, and so I don't feel at all entitled to their documentation. If I'm working with Red Hat I'll be working with someone who has a support contract. But if I were, for example, someone involved with Fedora who had worked on resolving issues in Red Hat, I would feel like Red Hat was acting improperly, regardless of legality. Are you arguing that it's okay to free ride on the work of the community and not give back what you create? Stop muddying the waters by flitting back and forth between morality and legality, please.

You keep using that word 'open source'. I do not think it means what you think it means.

Devil's advocate: software isn't free or easy to create either. Maybe open source license should require "best-effort" documentation, or prevent the witholding of it.

It's sound business because I'd rather RedHat continue to be the leader in the space of commercial open source than Oracle. We've seen what Oracle does to open source since they acquired Sun and it's not pretty.

Though do note that Oracle Enterprise Linux existed before they acquired Sun.

But not before RHEL 5. So, they made this change with their next major release RHEL6.

Actually Oracle started copying RHEL4 as Oracle Unbreakable Linux, but it wasn't a threat until the RHEL5 variant.

Red Hat is under that obligation to give out their source code. They are under no obligation to explain it to you. For that, you have to pay them, or figure it out yourself. I have no problem with that model. Why should you feel entitled to their knowledge base or documentation if you haven't paid for it?

IIRC the GPL does, actually, require you to enumerate your changes to the code. This doesn't necessarily have to be in the form of a diff, though, and can be in a ChangeLog instead.

You're probably thinking about this bit:

2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions:

a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.)

Seems like all it requires is that you say "We changed this file, and we changed it on this date." I suppose one could question whether giving detailed information about the changes is simply the "spirit of the rule" or if it's actually "going above and beyond."

What it comes down to is that paywalled documentation is, in my experience, universally inferior to documentation that is publicly available. In my last job I saw "Proprietary Information of <company>" stamped over all the documentation I used, and the thing was the most insulting thing I'd ever seen. The "information" was less useful than the reverse engineering my colleagues had done over the years.

Red Hat's documentation is better now, but documentation tends to rot when people can't see it until they're in the midst of using it. I feel entitled to the knowledge base because I'm not paying for something unless I have a good idea of the sort of issues I'll have when using it, and what the company does to resolve them. Red Hat has made a fine business respecting this up until now.

On the other hand, a lack of publicly available documentation is less important when source is available.

Uh,

If, hypothetically, they were able to close, to own, the source they were dealing with, it really wouldn't matter how much they'd invested in the previously open source. It would be just source they invested in 'cause they wanted to sell it and there would be nothing "altruistic" about it (and no benefit for any who didn't pay them).

So, no, they can't compensate for withholding information by investing in open source. They might currently be only withholding an "acceptable" amount but their investment in open source and their information-withholding can't be naively compared to see if they "balance out".

Isn't this kind of pointless though? On a previous HN post about this, someone said that you could just use diff to extract the changes. It won't be long before someone creates a site with the patches that Red Hat should have provided.

That would make "clean" patches, meaning you get no real history of why things changed, which is important in understanding a large project like Linux.

You need to change to adapt to things. Do you do things exactly the same as you did in high school? Probably not -- you adapted to the changes that life threw at you.

RH is a small company that has to compete against behemoths with a full portfolio of software. While doing that, they make meaningful contributions that benefit all. A company like Oracle buries the cost of their RHEL clone in a larger purchase, or wedges it into an ELA.

Why is it so unreasonable for Red Hat to force Oracle to do their own homework, especially where the bleeding edge development work is all openly available via Fedora.

I'm posting this to add a continuation to the story about Red Hat having changed the way they release their RHEL 6 kernel:

http://news.ycombinator.com/item?id=2272535

and

http://news.ycombinator.com/item?id=2283188

If anyone actually cares about this: disclaimer -- I work for Red Hat. Not on RHEL.

Props to Red Hat for not putting out a bunch of BS pretending like it's not really what it really is.

It is not enough to be PR 2.0 compliant, which requires ending of control over the message, but it does ensure that adoption is much easier.

Here's why Red Hat has to do this:

http://us.ft.com/ftgateway/superpage.ft?news_id=fto041820061...

Financial Times: Is open source going to be disruptive to Oracle?

Larry Ellison: No. If an open source product gets good enough, we'll simply take it. Take [the web server software] Apache: once Apache got better than our own web server, we threw it away and took Apache. So the great thing about open source is nobody owns it – a company like Oracle is free to take it for nothing, include it in our products and charge for support, and that's what we'll do.

And as open source slowly eats into their proprietary products, they have less and less to differentiate themselves from any other software company, and it becomes harder to lock customers in to their platform.

Oracle will have plenty to differentiate themselves from 'any other software company.' Even if all they are offering is support, they are a trusted brand name among middle-management.

I'm unsure what the benefit to Red Hat is in this case. All a competitor need do is become a "customer" by buying a copy of RHEL 6, and then Red Hat is compelled by the GPL to distribute all of their patches to that "customer". Seems like more of a PR gaffe to me, than an effective countermeasure against the competition gaining access to their code.

Redhat is not obligated to take on anybody and everybody as a customer.

I think this was discussed on LWN; if you get the patches from Red Hat and redistribute them, Red Hat will cancel your service contract. So it could work, but only once.

one fake company after another. So long as it's still cheaper for Oracle to maintain the fake companies, they'll do it.

Thanks, I don't keep up on LWN. It seems to me then, that Red Hat has created their own fork of the kernel. Am I overreacting?

That has been going on forever. I remember the days when Red Hat was shipping a "2.4" kernel that had all the features from 2.6 backported into it. They're still submitting the patches upstream, so if you want an "unobfuscated" kernel tree you can get it from Linus.

I think you're unclear on what is not being shared. From what I understand, old version of RHEL had a vanilla kernel, and then an RPM with RH patches to that kernel, as well as the documentation for those patches. New versions of RHEL will not have a vanilla kernel; it will only contain a kernel with their patches pre-applied. Documentation for those patches will only be available to subscribers.

The code is not being treated as the commodity, but the documentation for that code.

So, corporations with deep pockets (read Oracle) can buy the subscription through a proxy and have full access to the individual patches and full documentation. The rest of us independent and poor hackers cannot afford it and we suffer.

Yeah, good idea RedHat.

And yes, I know the GPL does not compel them to be nice.

Yea, SAP bought TomorrowNow which did exactly this. Fortunately they did it against proprietary software so Oracle was able to sue.

I have heard but not validated that RH contracts have a clause that if you ask for the source, they give it to you and then promptly cut your support contract.

The source is available: see ftp.redhat.com:/pub/redhat/linux/enterprise/6Server/en/os/SRPMS for instance.

Maybe explicitly asking for the source is considered different somehow from just ftping it, but I doubt it very much.

There is nothing wrong in what Red Hat did. I heard that some customers indeed bought oracle's unbreakable Linux, but switched back to RHEL after experiencing oracle's crappy support.

The wonderful thing about Free and open source software is support being not a monopoly. You don't have to depend on one person. Oracle's offering will make RHEL and its support team much better and competitive.

This PR statement unfortunately fails to address the knock-on effects to the CentOS project that this will have.

It was discussed in the CentOS forums, though: https://www.centos.org/modules/newbb/viewtopic.php?topic_id=...

To quote the forum thread:

The CentOS kernel is just a rebuild, so there is no problem there. In the case of the centosplus kernel, because it may add patches, some extra steps might be needed. But again, that is not a major issue.

Disclosure: I do work for Red Hat, but not on RHEL. Just posting info I saw earlier.

Would it not be possible to compute what patches have been applied?

I'd guess for many patches it'd be trivially easy to determine if they were in or out; for the rest a brute-force search could be done...

"for the rest a brute-force search could be done..."

Huhwhat? Brute force: A + B + C = D, where D and A are known.

I'm assuming the parent was talking about publicly available patches, not internally developed Red Hat ones. In that case, you could determine B and C out of a list of possible values by brute-forcing.

No, some of Bs will be known as they're public patches.

the remaining aggregate of C would be 'redhat patch to 2.6.x-y'.

CentOS doesn't give a shit, they just build the rpm. This is meant to frustrate outfits like Oracle, where they mix and match patches to produce a Red Hat+ kernel (a least, so Red Hat feels).

The thing is, Red Hat is free to grab Oracles changes and incorporate them back. Tit for tat and all that. This policy change surely must have come from upper management. Real programmers know it is no serious problem for Oracle, only an annoyance for everyone else. It is also very much against the spirit of the GPL.

I respect Red Hat for all the open source work they do but this change is lame. Piss a bunch of your most loyal customers and advocates off for no real gain. Sounds like a great plan.

"Red Hat is free to grab Oracles changes and incorporate them back."

Oracle does jack shit for Open Source software, including the Linux kernel. There is nothing useful for Red Hat to incorporate back from the Oracle kernels.

This is clearly a move to counter Oracle, and I can't blame them. Oracle has more money than god, and a willingness to do anything to win. They're rebuilding RHEL, rebranding it, and extracting money from their sizable corporate userbase for it...money that probably ought to be going to the folks who actually built the distribution, and the people who build the underlying software (Red Hat contributes more to Linux and Open Source than any company of its size, by a significant margin; Oracle contributes effectively nothing, in comparison).

That may be changing...Oracle has hired on at least a couple of reasonably well-known kernel developers (Chris Mason, for one). But, for now, Oracle is leveraging the development work of Red Hat far more than Red Hat could possibly leverage anything out of Oracle. Oracle just isn't a team player in this regard, and it's not built into their culture to become a team player, as far as I can tell.

I don't follow the logic. If Oracle does "jack shit" then how how does this monolithic patch inconvenience them in any great way? Also, with their money I'm sure they can find a way to split apart the patch again (either via Red Hat's web interface or by brute force). I guess I'm not management material because this move makes no sense.

Most of the value in question is in the information about the patches, and not the patches themselves. The explanation from Red Hat makes it clear they'll be closing up quite a bit of their back and forth discussions with customers about the bugs they fix and such (which I have a bit more of a problem with, actually; if a bug tracker isn't open to everyone, it's value decreases remarkably, including to the paying customers who are using it).

Oracle can certainly deal with it. Many problems are solvable with sufficient money. I just think Red Hat is trying to make it more expensive for Oracle to rebuild/rebrand RHEL while still remaining dedicated to supporting the upstream. I don't know if this is the best way to achieve that end. But, I have a great deal of mistrust for Oracle, while I feel pretty good about Red Hat. Oracle's handling of MySQL is not making me feel better about them, either, while we're on the subject.

(which I have a bit more of a problem with, actually; if a bug tracker isn't open to everyone, it's value decreases remarkably, including to the paying customers who are using it).

It helps RedHat focus on the bugs their customers care about -vs- bugs non-customers care about.

True, but you also don't get outside feedback/comments/help. You only get the paying customer(s) with the issue and RedHat developers.

> Oracle does jack shit for Open Source software, including the Linux kernel.

I think that's a bit of an overstatement. http://oss.oracle.com/projects/btrfs/

Red Hat:"Essential knowledge that our customers have relied on to support their RHEL environments will increasingly only be available under subscription."

FSF: "the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its user"

However necessary Red Hat might consider this move to be, it is just as clearly a move away from the spirit of the GPL however much it may conform to the letter of the GPL.

At the same time, open source companies need to compete on some basis and hiding information seems to be one basis. Is there an alternative to this? (serious question)

You have all of the freedom to change the code that you always had. The code is exactly the same, but instead of being released in small chunks, it's released in one big tarball. The commit splitting and commit messages could be considered documentation, they are not part of the code and not covered by any license.

The "Spirit" of the GPL argument has never rung true with me. It's not about creating shared projects and having big hippie hacker sit-ins. It's about keeping CODE free.

This is a perfectly legitimate move by Redhat. It's unfortunate that they felt they needed to do this, but they are under no obligation to provide source code to anyone other than those to whom they distribute, and the GPL certainly doesn't force them to support anyone at all - they want to support paying customers.

It's not a move away from the spirit of the GPL in any way.

Anyone else see this?

"redhat.com is temporarily unavailable. Please try back later."

I was starting to wonder if the controversy surrounding this had taken their site down but the article is still there, a few screens down.

Sustaining an artificial scarcity of high quality integrated OS releases has been what RHAT has been about since the beginning. Why is this new move any surprise?

I wonder which of Red Hat's actions make you think that they're trying to create an artificial scarcity. There are hundreds of other ditros. How does Red Hat keep them from creating a quality product?

By not sharing software work that is easily shared. They love to draw from upstream but they share only those parts of their infrastructure that don't threaten scarcity.

Why not list actual complaints? So far you've just been making generalizations. What software are they not sharing? Are they violating the GPL?

The site seems to be down..

Hmm, sounds like Redhat is really looking for a BSD type license.

And that's how you make yourself irrelevant....

TL;DR: We found a loophole on the license, we are going to abuse it , Problem ??? :trollface:

> Red Hat often talks about upstream first, the practice of openly developing kernel features and bug fixes as part of the most recent upstream kernel before we ship them in Red Hat Enterprise Linux. We know the value of getting code open from day one, debating it in the public forum, and letting it mature through a cycle long before it reaches our customers’ data centers. As the kernel community is well aware, it is standard practice for Red Hat to submit fixes that we find in supporting our customers.

[...]

> Why did we make this change? To speak bluntly, the competitive landscape has changed.

[...]

> but rather only to our customers who have recognized the value of RHEL and have thus indirectly funded Red Hat’s contributions to open source that will advance their business now and in the future.

Translation : RH believes in the shared source model as initiated by MS, and not in the free software model were knowledge is valuable and should not be hidden, and were new advanced are done on the shoulders on giants.

From now on, I consider RH as a traitor for the free software community, and will handle it like that (unless they change their unacceptable behaviour)

You are a bad translator and your attitude is perplexing. RH is conforming 100% to the GPL.

Reply on news.ycombinator.com