“Security problems are primarily just bugs”
blog.erratasec.com
blog.erratasec.com
Sorry, but I completely disagree here. Many of us that care about security are developers that have to clean up the mess of such ideas.
And to once more paraphrase Hoare's Turing award speech.
"Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to--they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law."
Linus describes two choices: 1) add warnings now to give developers a year to fix security issues, then add panic 2) skip warnings, add panic now, and start crashing everything for all users.
For example the chance of a kernel bounds error leading to an actual kernel security breach is vanishingly small in practice. This is because the vast majority of kernel bounds errors are just bugs, not actual kernel security breaches that harm an actual user. Whereas an immediate kernel panic will harm a user immediately 100% of the time.
If you prefer to run an immediate-panic system, or you have a different kind of threat model, then you can choose to catch the warnings and do your own panic, alerting, shutdown, etc.
Things which intentionally destroy a running instance of Linux, or in some way render the kernel less valuable to the end-user really, really piss Linus off - in this context.
I believe this is a key aspect of the Linux evolution: the only people of any value and import, feelings-wise, are the users. Not the developers, not security experts, and so on. He has done everything to ensure that the focus stays on working code, and not feelings, but .. when he has to .. he sure climbs the tree and jumps hard on anyone trying to cut into this substance of the Linux zeitgeist. Make it valuable to the user, or gtfo.
But in the case of the kernel, there are millions of lines of code out there which depend on the vulnerability in order to run.
While it may be superior to leave the checks on when the system already runs fine with them in place, it's not the same thing as "throwing the switch" when they aren't being used, and just waiting to see where the dust settles.
Here "killing" buggy code means literally "kill -9" it. Is this really your job?
We don't have a 100% SLA for uptime. We can be down for a bit and not have broken any of our commitments to customers. (My current job's SLA is super lenient, actually, and outside business hours, the control plane can be down for a while as long as the services being managed are up.) We do effectively have a 100% SLA for security, because there's no way to reset the world to an un-hacked state.
Or to put it another way: Let's say it takes one year to straighten out all those cases. What should happen during that year? Linus' option is that the kernel keeps running like it has been running the last 20 years. Linus is against the option that all those cases crash straight away.
What you wrote about your code is that you can make sure to avoid those bugs in the first place, by putting in bounds checks. But that's not the position Linus is in. Linus has to deal with the past 20 years of lack of such checks.
I understand Linus' position about userspace backwards compatibility. I don't think anyone actually takes advantage of it in practice to the extent he seems to believe. That's why Red Hat maintains extremely long-term forks of the kernel from an ancient stable release, and even distros that do not make or lose money on compatibility (e.g. Debian) don't update kernel versions in stable releases. I've worked at what amounted to a Linux-on-the-desktop startup, and we were afraid to upgrade the kernel without upgrading a pile of userspace (udev, X, libGL, etc.) in concert - and when Ubuntu announced https://wiki.ubuntu.com/Kernel/LTSEnablementStack we were quite happy because it meant we could stop upgrading to every non-LTS Ubuntu release just to pick up hardware support. I've worked at a high-frequency trading shop that was hesitant to upgrade even point releases without significant vetting. These are both places where we hired people specifically for kernel expertise and where understanding the Linux kernel was part of our competitive advantage; it's hard for me to believe that the average company doing webdev on their Macs and running production on some random Linux AMI is more confident in kernel upgrades.
So, if you have some worry that your kernel that's been serving you for the last 20 years will start crashing instead of corrupting data, there's an easy answer: stay on your stable branch, which you're probably already doing. Make your kernel static enough that the argument makes sense. I would firmly agree with an argument that new security hardening techniques do not belong in a stable branch. Go run some test traffic on the new kernel, but keep the old one in production while you do that.
But I don't think anyone is served well by the advice to go ahead and take a new kernel release and trust that it won't corrupt data just because previous kernel versions haven't done so for the past 20 years. Or, for that matter, that the new kernel release won't start killing your processes. A misconfigured OOM killer is just a bug, too.
If it is vital to your application to detect a certain kind of bug in production, then turn the checking on in production and pay the cost. Maybe it is not vital to my application or my application does not have these bugs. In my case it would be needless to pay the cost.
To me, essential to delivering a good product is detecting and killing defective code before delivery. I recognize however, that this attitude is becoming more and more old fashioned.
You should benchmark this. It is not true in the general case that any safety net will certainly have a cost.
Here is a really good example of someone actually enabling checked arithmetic in production, benchmarking, and finding no measurable change in performance: https://www.reddit.com/r/rust/comments/2nlis8/which_classes_...
Given that memory access is much slower than an integer comparison within the CPU, it's entirely possible that a bounds check can have zero performance impact, if it runs while the memory is being fetched, and your code is generally correct (i.e., does not cause the bounds check to raise an exception most of the time).
Premature optimization is the root of all evil.
(And if you're really interested in performance at the expense of security, consider running a unikernel: a lot of people run environments where there's essentially a single user on the box, and therefore no effective privsep. So why mess with copying things between userspace and kernelspace, enabling hardware protected memory, etc. when you don't need to?)
The problem is that you cannot prove the non-existence of bugs, even if you formally verify all your code, you've only proven that it conforms to spec, not that the spec is flawless.
This is why you should always code defensively. Assume all code is buggy and take steps to deal with that. Code is written by humans, and humans make mistakes.
That doesn't mean that you shouldn't try to deliver a flawless product, only that you should acknowledge the fact that you're going to fail.
To explain: your spec will have a few hygiene clauses like 'no stack overflow' or 'no crashes' and some things about what useful things the program should actually do.
The latter is hard to formalise and deserve defense in depth. Lots of the former are easy to formalise and you can rely on the compiler to get it right.
And that's what we are doing already anyway: even in C we just trust that a loop will compile to the appropriate conditional jumps and don't add defensive measures to check that.
And no, you can't rely on the compiler to get it right. Maybe if you only write Haskell you can get there 99% but there will always be human errors in code that the compiler can't find.
Yes, in practice you seldom have much of a formal spec, agreed.
This sounds fully reasonable to me. If you're an OS builder and prefer an immediate-panic hardened system, then catch the warnings.
Linux kernel post: http://lkml.iu.edu/hypermail/linux/kernel/1711.2/01701.html
"Some thoughts on security after ten years of qmail 1.0"
It's a bit easier for me to read without a fixed page layout.
Of course, I put a fix and a database migration in place as soon as I could and all is well now, but this worries me. It worries me because it must have been done out of ignorance (bad) or intentionally (worse) and billed as a feature or something. Neither of these things are mere bugs. This gaping flaw wasn't introduced by accident.
So when they wrote a script to run a db extract, get all company salary data in csv and share it with literally 50+ people I gave up. Getting them to even encrypt data seemed impossible.
Not the GP, but I've experienced similar things during my career, and these sorts of analogies tend to work really well when explaining technical points to non-technical (and, sadly, even "technical") people.
In an analogy with construction, each room is a functional requirement, and non-functional requirements are the materials you used to build, which can be considered some sort of quality attribute.
And just like in construction, if after building a house you are given more strict seismic requirements, fire prevention, etc... you might need to rebuild the entire house.
You can turn non-functional into function requirement if you specify a boundary. For example, a 60 frames per second requirement is functional. For security, you can require "secure under certain attacker models" to make it functional.
Example: Is Signal communication secure, if you assume the attacker can only read data on the server? There is an answer and it is probably "no". That does not mean, Signal is secure in general. An attacker which can access your phone breaks the security.
(It is also not really binary, because someone might find a hole somewhere in the future, but for practical purposes, we can assume the crypto holds)
Not so easy.
1) Very often (too often) functional requirements are poorly worded, and they leave things open to interpretation or to implied meanings.
Example:
"as user Foo, when authenticated in I should be able to open and modify all the documents I own, so that I can amend them".
Does such user story look good? Probably; but it says nothing about the "contrasting" stories. A lot of times there won't be a story that says...
"as user Foo, when non-authenticated I shouldn't be able to see any document by anybody". or
"as user Foo, I should be able to see and modify my documents and only my documents"
so if a non-authenticated user or a different user has access to your docs, you aren't probably breaking a functional requirements, but it's probably not how the system was meant to behave by most stakeholders.
2) Very often the functional requirements are OK for the sweet spot, but break on corner cases; so many people just won't notice what's wrong.
I would disagree with this assertion. A really common phrase in the infosec community is that "Security is not binary"
https://docs.google.com/presentation/d/1rXyl_YF-0lg3W8yY9mSo...
Security problems are, first and foremost, about CORRECTNESS. If you find an issue, most probably something has been done in an incorrect way by somebody who didn't really understand something. It's much harder to find out some security bug which is not a functional bug (for some part of the domain, not just the sweet spot).
Of course that doesn't really apply to C or C++, where it's easy to do dumb mistakes by chance.
"Nowadays I am much more insistent on programming language support for smaller-scale partitioning, sane bounds checking, automatic updates of “summary” variables (e.g., “the number of nonzero elements of this array”), etc. By “sane bounds checking” I don’t mean what people normally mean by “bounds checking,” namely raising an exception if an index is out of range; what I mean is automatic array extension on writes, and automatic zero-fill on reads. (Out of memory? See Section 4.2.) Doing the same work by hand is silly"[0]
Fundamental difference of opinions like these is why forks should happen in the open source world. I would like to see someone do a "LibreSSL" version of the Linux kernel, by cleaning it up of all the unneeded legacy cruft, modernizing its architecture, and making it more secure.
I imagine only someone like Google (for servers, Chrome OS, and Android) or Microsoft (cloud services) could take on such a project, but of course they could only lead such a project. They would also need an alliance of partners to support the project.
However, if they are committed to it and serious about making it way better from a security point of view, I could see many companies jumping ship from Linux Classic, especially in the automotive and IoT worlds, also also from web hosting world, and so on.
Alternatively, perhaps the large companies could start supporting Rust OS/kernels such as Redox.
Is this a legitimate leadership technique? I mean I presume the intention is to have a kind of megaphone which will get the attention of a widely dispersed, highly independent group of people. The ultimate cat herding weapon. On the other hand I find it really unpleasant and feel like there must be better ways.
TL;DR: It's okay to yell at someone if they're being stupid, but you need to be sure.
"This is wrong, you've introduced a bug here. Go fix it."
https://en.wikipedia.org/wiki/Dog_behavior#Leadership.2C_dom...
It's not a "technique". It's Linus' personality. One can go back to his Usenet postings from 1991 and see he's been writing like that when he was not a leader.
The problem that people are having with Linus' rants is that he makes the receiver uncomfortable and offended. I have been on receiving end of such rants. It feels really shitty, and I would love never to be subjected to something like this.
But the point is, it's directed towards a person who is about to significantly worsen the quality of a core information tech that is used by billions of people around the world. Thousands of human-hours will be lost. Millions of people affected in one case or another. And quite probably, in a couple of cases something really bad will happen if a process that previously worked flawlessly suddenly quits. Isn't Linux used anywhere in critical infrastructure? Hospitals? Domes? Banks? And it's not a random contributor, but (as I understand it) a major maintainer - a person who has a lot of authority and responsibility that goes with it.
So yeah, I think that this person deserves to feel shitty for a bit.
Perhaps it is for the Finnish. Check out "management by perkele" https://infogalactic.com/info/Management_by_perkele https://en.wikipedia.org/wiki/Finnish_profanity#perkele
I don't really mind people swearing, but attempts to frame that as something noble when it is nothing but emotions targetting argumentation technique are ridiculous.
This approach also doesn't work in modern professional armies note that from the beginning SF Troops (starting with the SAS) are very informal.
Bigotry of low expectations is something like saying voter ID is racist because black people don't have voter IDs and are not reasonably expected to be able to obtain them.
Sure. But he knows that, right? In which case participating in an international project with the extreme of norms in your homeland is not a "he can't help it" situation. It's no different from me saying "I'm not Finnish, but I really like Finnish culture, so I'm going to be an asshole." I'm still choosing to be an asshole.
> Bigotry of low expectations is something like saying voter ID is racist because black people don't have voter IDs and are not reasonably expected to be able to obtain them.
(This is mostly off topic, but the standard claim here is that the legal system makes it intentionally difficult for black people, or people who live in majority black areas, or poor people to obtain voter IDs. I don't think I've seen a serious argument that anyone actually thinks black people are racially less competent at getting IDs.)
I dare you to find European devs who believe terms slave and master in tech context are offensive and should be replaced.
I also saw a Swedish guy suggest (on a Stack Overflow Site) that some one with the first name Mohammed change this to his "White" name of Martin - they seemed totally oblivious of what they just sugested
Oh, hello, I'm a European dev and I'm making a concerted effort to remove those terms both from my vocabulary and any code / tickets / documentation I come across.
I think you'll find that the Americans don't have the only legacy of slavery to be worried about - they just kept it going longer than most. As a British person, I've got more than enough historical shame about slavery to make me want to avoid those terms.
What's with the Europeans or Americans believing all sorts of generalized falsehoods about their counterparts ?
Of course there are European developers who hold such beliefs.
"Primary" and "replica" seem just as good to me.
You can argue whether this development model is a good one, of course, but I think it's quite clear that it's working as intended, rather than these being random outbursts.
What do you mean?
If by "legitimate" you mean effective, then sure; linux kernel development seems to be humming along and providing value to lots of people.
If by "legitimate" you mean socially acceptable, then I'd look to the large community that seems to find little problem with "Linus language".
If by "legitimate" you mean causing no harm, then perhaps you have a case. Linus does seem to get under some people's skin. But what are our priorities, really?
From a stoic standpoint, Linus' words have nothing to do with anybody but himself, so it says a something about me or you if we get bothered by him.
However you look at it though, Linus is a human just like you or me. If we dislike him, then we're free to ignore him and move on; why spend time trying to argue what he "should" be doing?
That's just a rephrasing of the old rhyme "sticks and stones may break my bones, but words can never hurt me." -- which has been shown to be wrong. Sure, you can go around and try to block everything away, but it will still take its toll.
I find it often pays to ask, "Hmm, why would I be bothered by that?" in place of saying "You should stop saying that."
That's actually an interesting strategy. I'm not sure it works for everyone, but I would at least consider it.
I have the same problem as Linus that I'll go overly dramatic and nasty in calling people stupid in replying online, something I don't do to people's faces. The more strongly I feel about something, the more likely it'll happen.
It's like some sort of social filter doesn't get applied, one I use when I'm speaking, but it just gets skipped when typing. When I read back those posts, I'm embarrassed, not proud of my "technique".
It's my problem, not the victim's. Luckily, because I'm not internet famous, those sort of posts get downvoted and I get some sort of feedback loop. But Linus gets defended and the cycle continues.
Here I'm mainly thinking of us on the recieving end. From this standpoint, by blaming Linus or whomever for our emotional reactions, we end up disempowering ourselves quite a bit. There's a lot less we can do about Linus than about our own emotions and psychology, which is why I think it's useful/skillful to introspect instead of blame.
With the roles reversed though, it's certainly more helpful/skillful to focus on how we can communicate effectively. Blaming others' actions/reactions is just as disempowering as before.
You're trying to dominate a social situation, because you think you are right (and by extensions, that there are others who will support you and strengthen your position) and because there's a lack of physical threat.
In the real world, you would not want to cause a ruckus, because there could be consequences, and your are not the dominant member of the group. For example; someone could punch you.
If you could, you would want to dominate any situation that you can. We are wired that way.
By what measure? Linus seems to be getting away with it.
Whether it's because his other qualities (or just the other qualities of the Linux project) let him do that and this style is detrimental or whether the style actually helps, is a different question.
There are many other rhetorical devices, raising your voice is not the only way to counter inattention.
You may want to read Linus' earlier messages in the same thread, such as the immediate predecessor, just as an example:
How many times you say, "I guess you didn't understand what I said; in this project it's not acceptable to sacrifice the user for developer convenience" before you "raise your voice. Some people seem to think the only acceptable answer is "never". (And if you do, there is a zero-tolerance policy where you kick out the developer in question, even if he is the founder.) That's not the Linux kernel, for better or for worse.
I'm not saying it's acceptable or not, just that I don't buy your claim that the tech industry is special in that regard, let alone that Linus is somehow the source of it.
I think you're grossly overestimating the impact Linus' rants have on the industry.
A previous employer had someone in that "role", I don't think he will ever be able to work at any company that employs a former cow-orker of his. It may be fun for a short while for a certain kind of person, but it's career suicide.
There are some roles, like the one Linux has, where a critical aspect of the work is to tell NO to people and reject most of their requests.
I see a bunch of github issues where Alice comes up and tell the developers their code is insecure.
Of course code is art, so developer Bob takes it to heart and say something along the lines "what do you mean? Can you prove it? I don't think I agree, the code is nice like this".
And then Alice and others start becoming aggressive, Bob starts getting defensive. At the end nothing is probably fixed but Bob is now sour and will probably become someone like Linus.
What I'm saying is that Linus must probably have had, and still has, to respond to a bunch of "f*cking morons" (to quote his words) and he's developed a thin skin has well as a wearable douchitude as a result of that.
Someone mentioned Usenet archive from 91 when Linus was just the same.
Nice assumptions though.