> We have been providing grsecurity freely for 16 years.
Meanwhile the kernel upon which their work is built has been provided for free for much longer, and continues to be.
It must be particularly irritating in Spengler's case, because he works in a field where most of the best people make millions turning poorer versions of the same ideas into commercial products that only huge companies ever get to play with, while he spends his career arguing with upstream and taking shit from people on message boards.
> The apparent inability (and perhaps more importantly - total unwilling[n]ess) from the PaX team to be able to see what makes sense in a long-term general kernel and what does not, and split things up and try to push the sensible things up (and know which things are too ugly or too specialized to make sense), caused many PaX features to never be merged.
In a comment [0] the PAX team says
> we never ever considered submitting grsecurity or PaX for mainline inclusion [...] anything similar in the past was user action, not on our behalf
I don't know how Spengler and PAX relate.
That stopped last year at the same time the -stable patches were made private.
Torvalds does great work. Spengler does great work. The kernel team does great work. The PaX team does great work. Torvalds is not always the easiest person to get along with. Spengler is not always the easiest person to get along with. Life is sometimes complicated. None of this takes away from anyone's achievements.
You see this back and forth happening over at Windows as well where continued backwards compatibility is weighed against future security.
Just look at the more restricted capabilities of anything WUP vs Win32 (never mind Win16).
Maybe they're just thinking "our customers value our work, so let's focus on providing value to them instead of to people who don't value our work". Whether or not that's true, I don't blame them.
https://unix.stackexchange.com/questions/59020/why-are-the-g...
The technologies these dudes developed are used in the vast majority of OSs outthere.
Instead of posting a cynical comment here you should thank them.
Also, pedantically, this wouldn't be a 'conspiracy theory', since all I'm suggesting is that one guy is being disingenuous on the internet.
https://unix.stackexchange.com/questions/59020/why-are-the-g...
To clarify some technical aspects, SELinux and grsecurity are not answers to the same problems, and they never meant to be. SELinux was always meant to implement things like multi-level security, bell-lapadula model, and extending 3rd party application with SeLinux roles, and the security daemon. Grsecurity has file oriented rbac system that was more approachable, and a variety of other patches (which are what grsecurity has been more known for - they have been always very excellent).
Knowing Spender, he was probably actually paid to stop. That's my guess. Not going to iterate on that.
https://www.linux.com/news/google-developer-kees-cook-detail...
He's been working on this for a very long time.
no, you're wrong. how do you think we developed our code? did all the 8+ MB worth of it pop out of our head all at once? or more realistically, did we develop the features piece by piece, not unlike how upstream linux is developed?
> They have very specific "chunks" of functionality which are quite invasive, by design.
define invasive. linux itself has 'quite invasive' features too yet that didn't prevent them from being developed and upstreamed, so not sure what you were trying to imply here.
> This goes against the model the Linux kernel is developed, so the two development > communities are simply mutually exclusive.
this narrative only exists in your head, not in reality. our work is as much upstreamable as any other kernel code that went in over the years (how else do you think some of it could get in already?), it's just that it can't be done in one's free time.
Did I get that wrong? From reading Brad on LWN for literally years wank about Linux security (he's absolutely not wrong to complain), this was my take. Or it could be summarized as (this is satire):
PaxTeam: This is our code, it is open source, it is more secure than the dumpsterfire that is upstream, take it if you want it.
Linux Upstream: That is nice, now break it into 1000 individual patches each independently git bisect-able. Also, can you change the interfaces of these design flaws in Linux and THEN entirely re-do your well tested patch to match the way we do code upstream and match the interfaces we would *like* Linux to have that we're going to expect you to code for us.
PaxTeam: No, we want to do more interesting things like build more secure patches for Linux and look at pictures of cats on the internet, as it was designed to be used for.
Linux Upstream: ...as for your satire, it's just got one thing wrong, but that kinda kills the rest of it: we never said "take it if you want it" and thus we never embarked on the rest of that journey.
That's not an argument against the current state of the pax/grsec being much more monolithic in architecture than the Linux kernel (I don't know if it is; just that that argument doesn't prove it's not.) Sure, the commits are small. 1000 small commits can add up to 100 small+simple components, or to 10 rather large+complex ones.
> define invasive
"Invasive", I believe, refers to how much of the kernel needs to be modified to upstream any given feature (regardless of patch size.)
The opposite of "invasive" is "isolated". 1000 lines of code in one module representing a new USB device driver, for example, is very non-invasive. It doesn't even need to be "hooked into" anything; it just puts a vendor+device ID into a database, and some existing logic will then detect devices with such IDs showing up on the bus and load the module in response.
Meanwhile, 1000 lines of code that make breaking changes (however small) in the ABIs or preconditions/postconditions of 100 different existing kernel functions, is an "invasive" patch. It requires a lot more testing than the USB driver to be accepted, because those 100 existing functions will be executed by a lot more people's machines than the USB-device module (which will only get executed on machines that have that device.)
> The problem is that getting things into small individually testable components > is literally anathema to the Pax/grsecurity model.
nowhere does this talk about what the patch looks like. it talks about "getting things into small individually testable components" being "literally anathema to the Pax/grsecurity model". i.e., that decomposing the monolithic patch into composable pieces is somehow against our development model whereas said monolithic patch was developed exactly this way (which means the reverse process would reproduce these 'testable components'). that argument (my statement) directly contradicts Jeff's so only one of them can be true and as the author of the code in question i know which one is what ;).
as for invasiveness, you're only stating the obvious that there're more and less complex pieces of code yet both kinds (well, all kinds in the complexity spectrum really) can make it into the kernel. that is, this argument cannot be used against incorporating grsec on its own.
You asked for a definition, not an argument. Definitions are supposed to be obvious. Not every line of a reply has to be a rebuttal, just because one part is. :)
> which means the reverse process would reproduce these 'testable components'
The reverse process of how you built up the system (by committing changes), would break the the patchset down into commits. Commits might be patches, but they're sure not components.
A commit can change a line in the middle of a function in the middle of a random file anywhere in the kernel. A component is its own file/module/function that can be independently tested and verified and, most importantly, judged for how well the design-choices in its implementation solve the user-story they purport to solve, compared to other possible implementations.
A random one-line commit can certainly implement functionality—or can fix a bug—but it can't (usually) introduce a user-visible feature on its own. A series of one-line commits can, together, introduce a user-visible feature; but if they do, they need to be merged together into a single patch so that the design they implement can be seen, in order to be evaluated.
If you have take your codebase and just export all the commits as patches, each one might—as it probably originally did—just add one or two lines to the body of some function. But if all the commits necessary for a given feature end up combining into one big, monolithic blob of a patch—or end up extending several functions in the kernel into being large, monolithic functions—then the design of the feature is probably bad, and the patch will therefore probably be rejected.
What the kernel maintainers expect, in such a case, is for you to take your features and refactor them into independent subfeatures, such that each "patch that implements the feature" is fairly small. They expect you to do the same thing to your patchset that is done in a web-app when it is refactored into microservices: to make each component small enough, and non-interdependent-enough, that an argument over the design of the patch can involve each invested party writing their own version to demonstrate their idea; and where each component can be easily ripped out and replaced with one of the alternate designs, without affecting any of the other components.
For a good case of this already happening, look at OpenVZ. OpenVZ was a big, monolithic patch, that was never going to get upstreamed. It was refactored into the individual components of cgroups, the CPU freezer, memory quotas, and the addition of a number of namespace hierarchies. All of these components "add up to" something roughly equivalent to OpenVZ, but they're not OpenVZ. (In fact, they're effectively Docker; upstreaming them basically killed OpenVZ!)
vs.
> Commits might be patches, but they're sure not components.
make up your mind, are components (whatever that means for you) composed of commits or not? :) as for commits vs patches, all commits are patches + description, but we use those terms interchangebly in the linux world with this understanding. in any case, i stand by what i said, we develop our code the same way upstream linux (or really any sane project) does. we're just not developing it for the purpose of inclusion, maybe that's the source of your confusion.
Other people have gotten large changes into the kernel before, but they require interfacing with the kernel devs and agreeing to certain changes first. Nobody's going to accept a patch just because Google wants it for one of its products. And Google isn't going to change its product patches just so they can fit in mainline.
To give you an idea of Kees's contributions:
~/git/linux (master)$ git remote -v
origin git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (fetch)
origin git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (push)
~/git/linux (master)$ git pull
Already up-to-date.
~/git/linux (master)$ git log --author="Kees Cook" --pretty=oneline | wc -l
701
And do you have proof that PaXTeam is not paid, if you are indeed PaXTeam?
you're wrong, see item 6 in https://opensource.com/article/16/12/yearbook-9-lessons-25-y... though you might want to demand that Greg prove his identity first ;).
> And do you have proof that PaXTeam is not paid, if you are indeed PaXTeam?
sure, come visit me in hungary and i'll take you to the local tax authorities and grant you access to my files. of course you'll have to prove your own identity first ;).
>> sure, come visit me in hungary
That'd be amusing, maybe sometime this summer? I'm kinda busy right now setting up my business :')
Let me know when you're around the Benelux, we could do the key-siging song-and-dance over a coffee or beer.
As for grsecurity, 100% of the core grsecurity team (that work at "Open Source Security") are paid for their work. I will admit that I'm not fully aware of the interactions between PaXTeam and grsecurity, but I'd be shocked to hear that nobody at PaXTeam works at a software company related to security or kernel hardening.
> As for grsecurity, 100% of the core grsecurity team (that work at "Open Source Security") are paid for their work.
that's 100% false. both spender and me are developing our code in our free time. what the company is for is customer support, not R&D. shocked you are? :)
GregKH, who you linked in a cousin comment. IIRC ~20% of code authors are not associated with a company. And if we go by your logic, then an even larger proportion are not "being paid for their kernel work". For a concrete example, I'm a maintainer of container runtimes at my current job but I have contributed code to Linux as part of my job -- does that count as "being paid" for it? In my mind, yes. In your mind, clearly not. But in GregKH's statistics I count as an employee of a company (not an independent).
But since you're too lazy to look at your own link, here's the article for 4.11 (https://lwn.net/Articles/720336/). 14% of changesets and 13% of lines changed are by people not associated with a company.
> that's 100% false. both spender and me are developing our code in our free time. what the company is for is customer support, not R&D. shocked you are? :)
"I work in an L3 support role on $technology, but any R&D work I do on $technology is completely unrelated." It's like you're not willing to acknowledge that the only reason someone would pay a two-person team for support on a kernel technology like grsecurity+PaX is that the same team is developing it. So even if your invoices don't have "development" written on them, the only reason you'd have customers is because of the fact that you are the main R&D behind what you're supporting.
> The majority of developers are paid for their work[...].
that's not at all true for our case, that's all i pointed out.
> I'm a maintainer of container runtimes at my current job but I have > contributed code to Linux as part of my job.
if it's on company time (and thus dime) then yes, it's a paid job.
> 14% of changesets and 13% of lines changed are by people not associated with a company.
not really, more than half of each is 'unknown', so you can't tell one way or another. anyway, not sure what these are supposed to prove/disprove given what Greg himself said in the above quote.
> It's like you're not willing to acknowledge that the only reason someone would pay a two-person > team for support on a kernel technology like grsecurity+PaX is that the same team is developing it.
indeed it's not the only reason but since it's not your business (no offense meant just stating a fact), i can't comment on this further. what i did mean however is something different than the direction you veered off: our work isn't developed because it's paid for, it's a completely volunteer free time project (spender has a day job unrelated to this work, and until about a year ago i didn't have any at all in fact). that is, if you took the money out of the picture, our work would still continue to live on as it has for the previous 16 years. that is absolutely not true for upstream linux development (if it were then all these companies have been cheated out of their money they spent on developer salaries).
Sorry, it's 86% and 14% in Linux 4.11. It's literally in the link I posted. He gave a talk a few years ago at Linaro IIRC where he said "80% and 20%" as approximates but it seems like it's closer to 85% and 15%.
> not really, more than half of each is 'unknown', so you can't tell one way or another
Companies which pay their developers to work for Linux want to exercise their copyrights. It wouldn't make sense for them to conceal developers they are paying to work on Linux. I would argue more people who are listed as "from company X" are working outside of their work but have to declare their company ties due to IP worries.
> if it's on company time (and thus dime) then yes, it's a paid job.
Well, it's actually a bit more far-reaching than that. Most companies also claim that work directly related to your job (even not on "company time") is still owned by them (which you can assume means that's what they're paying you for). Maybe that's not the case in Hungary, but that is the case in America (and Australia where I am).
> if you took the money out of the picture, our work would still continue to live on as it has for the previous 16 years. that is absolutely not true for upstream linux development
I mean, Linux kernel development worked in this way for the first several years when it started. It doesn't really make sense to make authoritative statements about what is and is not possible in hypotheticals. Both Linux and grsecurity were developed for some time without direct income sources from most contributors.
All of the being said, I'm not sure why any of this is relevant. To me the statement "we are 100% unpaid for grsecurity R&D" sounds more like a semantic game than an earnest statement, given that you definitely are paid for grsecurity (whether it says R&D on the invoice is semantics IMO).
But w/e. You can pretend that you aren't paid for your work to seem more righteous. You don't get any sympathy points from me though.
you first quoted 80% and chided me for not reading the article linked to because it supposedly implied/had it in there. now you're moving the goalposts and point at a different article that i can't have guessed before you posted it.
> Maybe that's not the case in Hungary, but that is the case in America (and Australia where I am).
it's not necessarily the case there either. i had worked in both countries and always negotiated contracts that wouldn't have such overreaching clauses.
> I mean, Linux kernel development worked in this way for the first several years when it started.
exactly and you can see how much (or little, in this case) that achieved vs. what money did when it started to pour in. it's not at all hypothetical that if currently paid developers were stopped getting paid then the current pace of development would stop to a crawl (related example: look at gcc vs. clang/llvm after google/etc moved their developers from one to the other). easy test: would you continue to work on linux with the same pace/effort if your company stopped paying you for it? yes/no? if you answer yes then i also expect you to pay them back any past salaries you cheated out of them ;).
> Both Linux and grsecurity were developed for some time without direct income sources from most contributors.
in our case, it's not 'some time' but 'all the time'. that's the big difference which puts the original statement into a very different light:
> Meanwhile the kernel upon which their work is built has been provided for free for much longer
that 'for free' isn't at all free (money makes it happen) unlike our volunteer project (money doesn't make it happen).
Sorry, I assumed that 80% and 86% were close enough that a reasonable reader would be able to see that I had mis-remembered the second significant figure for statistics I heard a while ago.
My apologies.
> easy test: would you continue to work on linux with the same pace/effort if your company stopped paying you for it? yes/no?
Yes (though my work is not generally kernel work, I would still continue to contribute to the free software projects I currently work on at the same pace).
> if you answer yes then i also expect you to pay them back any past salaries you cheated out of them ;).
... why? An employer pays you to solve technical problems that rise from their business. That doesn't mean that as an individual I wouldn't work on similar problems anyway, it just means that I get paid to work on specific problems rather than whatever I find important.
Effectively an employer pays you to change your priorities to match your employer's priorities. How much your priorities actually changed is not relevant.
Let me ask you a question. If one of your customers found a security issue in grsecurity or found that one of the features of grsecurity was broken, would you prioritise fixing it over whatever interesting feature you were working on "in your spare time"? If yes, then congratulations you're paid to develop grsecurity. If no, then I wouldn't pay you for support because I would have gained very little for my support contract.
> in our case, it's not 'some time' but 'all the time'.
You can continue to claim that, and I will continue to call bullshit. While you might be able to argue semantics and say "technically we never were paid for any particular features" I find such discussion disingenuous.
> puts the original statement into a very different light:
That statement doesn't say "developed for free" it says "provided for free". If it said "developed for free" I wouldn't agree with it. But by the same token I don't agree that there isn't a significant proportion of development that is not paid, and I don't lend credence to hypothetical predictions about how Linux would be developed without anyone being paid.
Despite what you say, Linux was developed for free for the first year or so and was mostly developed for free for several more years.
vs.
>>and the source of those numbers is...? >GregKH, who you linked in a cousin comment.
the article i linked to has neither number.
> An employer pays you to solve technical problems that rise from their business
not at all. an employer pays for stuff it can't get done for free. it's basic economics at least but i'm sure some shareholders would also have trouble understanding why the company would waste money that way. you just admitted that you'd gladly do the same work for free yet you somehow failed to tell that to your employer who are thus paying you for something that they could get for free and spend that money on other important things instead. that's doubly damaging to your employer.
> Let me ask you a question. [...]
it doesn't have to be a security bug and it doesn't have to come from a customer, we look at them with the same attention and priority regardless. money has nothing to do with it, we do it (and we have done so since the beginning) whether we're paid or not. i think you just have a hard time imagining that what customers pay for isn't R&D (though we're open to such contract work too).
> That statement doesn't say "developed for free" it says "provided for free".
yes and it's disingenous as to be able to provide linux 'for free' someone has had to pay for it which is very different from our situation.
> I don't agree that there isn't a significant proportion of development that is not paid[...]
you're arguing semantics ('significant' vs 'majority') with Greg at this point, i'll let you two work it out.
> I don't lend credence to hypothetical predictions about how Linux would be developed without anyone being paid.
fortunately you don't need to predict anything, just compare the code of linux 1.0 and 4.11.
> Despite what you say, Linux was developed for free for the first year or so and was mostly > developed for free for several more years.
i never said anything like that unless of course you can quote me back on it.
anyway, you're clearly more interested in insults and ad hominem than a rational discussion, so you can have the last word.
Were you involved in my hiring process? Are you my manager, or a stakeholder in my employer's business? How could you possibly know that I failed to tell my employer that?
> i think you just have a hard time imagining that what customers pay for isn't R&D
I don't have a problem imagining it, I just don't believe that your customers see it that way. If I was a customer of yours, I would expect that more R&D work would happen because I'm paying you (mainly because I firmly believe in free software businesses and that such R&D should be paid for).
> (though we're open to such contract work too).
Heh, so it's not 100% unpaid after all? ;)
> just compare the code of linux 1.0 and 4.11.
Wow, code gets better over time? Holy shit!
> so you can have the last word.
Gladly. :D
Security improvements to Linux are great. However a hardened linux kernel is a derived work of the kernel.
It's unethical to take free software, build upon it, redistribute to customers but under contract agreements which prevent them from exercising their freedoms afforded by the license of the kernel.
If you think the linux kernel is crap, you are more than welcome to write your own kernel.