I'd also like my kernel to halt whenever an assertion does not hold, for the sake of keeping my sanity; not just for security.
Why would you not want this?
in that case, it will crash with a SIGSEGV sooner or later anyway
> We're trying to be secure against attackers.
We also want a browser that doesn't crash.
Most users are using computing devices a means of getting stuff done. They don't want to spend any energy thinking about how their software works, they want their devices to be invisible, which they use to run their Apps uninterrupted. The trade-off is whether to let Apps continue running vs hard crashing and taking down all the work they've done and all the mental energy and focus invested up to that point. If their Apps frequently crash most users aren't thinking, well I'm super glad the hours I spent on this paper I'm working on is now lost, the phone calls to my loved ones or movie I'm watching are abruptly terminated because someone's policy on hard crashing when a bug is found has been triggered. Their preferences and purchasing power are going to go towards non user-hostile devices they perceive provide the best experience for using their preferred Apps without any need for pre-requisite knowledge of OS internals.
There's not a single computing device that frequently crashes as a result of security hardening that will be able to retain any meaningful marketshare. Users are never going tolerate anything that requires extraneous effort on their part into researching and manually applying what needs to be done to get their device running without crashing.
Keep running your app although integrity corruption within the application happened is putting user data at risk. IMHO an application that corrupts 3 days long presentation file save is to every user more frustrating than the one that crashes due to error leaving you with 5 minutes of unsaved changes lost.
Microsoft have invented "Application Recovery and Restart" exactly for this purpose.
If user data is continually backed up to a remote site it's not going to be at risk from a local bug is it? Bugs exist in all software, Users are going to be be more visibly frustrated from their Apps frequently crashing then the extremely unlikely scenario where a detected bug corrupts their "3 days long presentation". They're going very unhappy if the cause of their frequent data loss was due to a user-hostile setting to hard crash on the first detectable bug.
> Microsoft have invented "Application Recovery and Restart" exactly for this purpose.
From Microsoft website:
> An application can use Application Recovery and Restart (ARR) to save data and state information before the application exits due to an unhandled exception or when the application stops responding.
- https://msdn.microsoft.com/en-us/library/windows/desktop/cc9...
i.e. restarting Apps due to "unhandled exception or when the application stops responding" in which case the App is in an unusable state and ARR kicks in to try auto recover it for minimal user disruption. The focus on providing a good UX, not a miserable crash-prone experience where users use their devices in fear that at anytime anything they're working on can be terminated abruptly without warning.
Data corruption - reading or writing corrupted data - files cannot be read, saved files get corrupted, API calls from/to external applications/systems fail or pass incorrect data Rendering problems - corrupted images, incorrect colors, improper content encoding, visual stuttering, audio deformation, audio skipping Input/output lags - unregistered kaystrokes, missed actions and responses to external events, mouse stuttering and misbehavior Improper operation - inconsistent results - repeated rendering yields different results (html), formulas/calculation results in data is inconsistent (excel, DWH) Access violation - access gained to invalid or protected areas - unprivileged access, license violations, access to areas protected by AAA, data theft (SQL injection, database dumps)
and others. If I figure out the application I'm using (web-browser) allowed a hacker to steal my data he would not have otherwise access to I would be more pissed off than if it crashed and I found an error about it in system log.
some people just use computer to do stuff to them there is little difference between "i lost my work because of a bug" and "i lost my work because of a security policy"; from a UX point of view both of them are the developer fault for releasing inadequate software.
Please leave out the uncivil swipes.
On topic: I can't recall now the details, but I read a paper once about a system which had no shutdown procedure at all, the only way to exit it was to crash it somehow or just shutdown the computer. The system made sure to save everything often enough and made sure to store the data in ways which allowed for restoring possibly corrupted parts of it on the next startup. This design produced a very resilient architecture which worked well for that use case.
The paper was from '80s or '90s, so it's not like we need to be in 21st century to design that way. I'll try searching for the paper later.
What you're talking about seems like crash-only with Erlang/OTP.
The OS[1] I've been talking about was primarily concerned with a single-machine environment, which resulted in slightly different design.
See also: "Crash-only software" https://lwn.net/Articles/191059/
PANIC on fault is exactly what you design into the systems.
The vast majority of processors sold are not i5 or i7 level but microcontrollers.
I'm firmly in the first camp (I'll take lock up and freeze thanks) but 99% of users don't care about a bug like that and just want the box to keep working.
Fail fast is a great philosophy for end-user software. But it is not that strictly good for middleware, and is almost certainly wrong for a kernel.
Crashing the whole kernel at the drop of a hat seems like a pretty extreme stance to take as a general policy IMHO. Killing and restarting the driver will usually suffice, although some data may be lost and have to be retransmitted.
Potential data corruption is far worse, so is a potential security compromise.
E.g. performing industrial control automation or airplane rudder control in a completely segregated network.
You want control over these tradeoffs, not hardcoded behaviors.
Look at seL4, minix3, eChronos instead.
you should panic in a production environment and reset the state of the machine (which has become indeterminate).
The correctness and validity of the data >> uptime.
If the remote end is going to ignore that data anyways, would it really be such a bad idea to keep running? Do you really want to go down in order to ensure that a remote who's ignoring your data can get correct data to ignore?
Of course you never know what sort of effect the corrupt data is going to have, so it's always hard to make that decision.
Like that issue with libraries linking against objective-c frameworks that has crept up with High Sierra and broke most of the Ruby world: Yes. The usage was incorrect. Yes, forking after threads are launched leads to undefined behavior, yes, knowing about it is a good thing.
But: So far the crashes have been rare in the common use-cases (or they would have been fixed), so High Sierra's change to blow up loudly when it detects the misuse has actually caused a lot of trouble for people where things worked fine before.
To the point where many Ruby developers were complaining about High Sierra "breaking" their workflow and recommending against upgrading.
The new check is totally justified though. The existing forking behavior was wrong and it could have lead to crashes down the line. It didn't though. And now people are forced to fix something that was never an issue to begin with.
It's a fine line to walk and while I generally prefer things to blow up as they go wrong, sometimes I catch myself wishing for stuff to just continuing to work.
On some self-reflection, I come to the conclusion that I want my cake and eat it too.
The kernel (for my intended usage) should intentionally panic only if there is a risk of corrupting my .tex files.
Systems that matter, are.
I was writing paper, on a PC, that was like "pip pip pip pip pip"
and then... like half of my paper was gone.. and I was like...
It devoured my paper.
It was really good paper. And then I had to write it again and
had to do it fast so it wasn’t as good. It’s kind of... a bummer.
What happens when a security bug stops the ventilating machine of a person lying in hospital bed, or halts the screen of a surgeon.
Not to mention voting machines, ISP's, telecoms.
For me having all those stopped, when properly exploited, looks more like a very scary DoS attack vector.
Imagine a security f*ck up, like Heartbleed, but this time with an option to halt kernels / systems.
As far as voting machines go, kernel panic sounds waaay better than executing malicious code.
> Imagine a security f*ck up, like Heartbleed, but this time with an option to halt kernels / systems.
IIRC heartbleed didn't allow you to execute code (it allowed you to read more memory than you should have been able to). A better example is every flash player bug ever. Would you rather that thing crashes or executes malicious code? Keep in mind that the malicious code can also shut down your system.
Also keep in mind that we're talking about userspace programs right now. This thread is about kernel bugs. Userspace programs already have the option to ask the kernel to kill them if they misbehave. A lot of them do that (using features like seccomp filter) and many more should. (Chrome and Firefox both use seccomp filter I think.)
1. 0-day in the kernel's USB code. You're part of stuxnet now.
2. 0-day in the kernel's USB code. You're part of stuxnet now. You also get a message that tells you how and where to report the bug that was exploited.
3. 0-day in the kernel's USB code. Your computer crashes. You're not part of stuxnet. You also get a message that tells you how and where to report the bug that was exploited.
Linus is wrong (it happens). Exploit mitigation techniques aren't debugging tools. They're exploit mitigation techniques. The fact that they also produce useful debugging information is secondary.
The IPv6 stack does a perfectly sensible and legal thing. The hardener code misunderstands the legal code, and causes a reboot.
That it was Linus is worried about -- often it is hard to tell the difference between "naughty" code which can never be a security hole, and genuine security holes.
They should all be fixed ASAP, but making code that previously worked make a user's computer reboot, when it is perfectly fine, is not a way to make friends.
That's not what Linus said though. What he said is:
> when adding hardening features, the first step should *ALWAYS* be
> "just report it". Not killing things, not even stopping the access.
> Report it. Nothing else.
> All I need is that the whole "let's kill processes" mentality goes
> away, and that people acknowledge that the first step is always "just
> report".
(Also, the quotes I chose don't really help me make my case but I don't want to edit now since you've already commented on it. His first mail is way worse: https://lkml.org/lkml/2017/11/17/767)
Basically what I'm disagreeing with is that exploit mitigation's primary purpose is finding and fixing bugs. That's just not true. Its primary purpose is to protect users from exploitable bugs that we haven't found yet (but someone else might have).
And you neglected a scenario 4: nobody is attempting to compromise my machine, but a buggy bit of USB code just crashed my system and took all my work with it.
Both 3 and 4 are mitigated by you saving your document often... it's not so bad, considering it can happen whatever you do.
Nowadays, Word is made to keep saving your change for that reason... They learned and designed it for the worst situation which is a whole system crash. If you can't handle that, well you aren't doing your work well.
Yes, of course we should save often, have decent backups, etc. But nobody is perfect and shit happens, and it'd be nice if the software you use didn't intentionally make it worse.
--
I pity the engineers working on ventilation machines and the like. Medical devices are insanely hard to get right; that's neck and neck with aviation testing. I'm reminded of SQLite3's "aviation-grade" TH3 testsuite, which apparently has 100% code coverage. Let's be honest; Linux's monolithic design can't really attain that.
I would never use Linux for a medical device. I say this as someone who just happens to only be running Linux on every machine in the house right now (and I have for years, it's just how things have worked out, it's not at all novel or whatever, my point is that I'm totally comfortable with it). I'd use L4 or something instead. In a pinch I'd use a commercial kernel with tons of testing. Maybe I'd even use Minix; I'm quite sure a lot of people in industry are seriously looking at it now Intel have pretty much unofficially greenlit it as a good kernel (lmao).
--
Voting machines, on the other hand; I'd totally use Linux for that, because the security/usage model is worlds apart. Here, I WOULD ABSOLUTELY LIKE FOR THE TINIEST GLITCH TO CRASH THE MACHINE, because that glitch could be malware trying to get in.
The user experience of a voting machine is such that you walk up to it, identify yourself, and push a button. Worst case scenario in this situation is that you do some involved process to ID yourself and then the unit locks up, so you have to redo the ID effort on another unit. That is, for all use cases, not going to be a problem.
(I think that's the first time I've used all caps in years!)
--
Telecom systems... those are also a totally different world. See also: Erlang. In this situation you would likely want a vulnerability to literally sound a klaxon on a wall, but have the system still keep going.
I'm reminded here of an incident where a country's national 3G system was compromised (not the US, somewhere else) by hackers and the firmware of the backend systems was hot-patched (think replacing running binary code - the OS allowed it, it was REALLY hard to even notice this was happening) to exfiltrate SMS messages and cause calls to certain numbers to generate a shadow call (which ignored mic input) to an attacker-controlled number as well.
Telecoms is a classic case of massive scale; nowadays a single telecom switch might be routing thousands of calls through at a time. Yeah you don't want even a single machine to go down. But you DO want VERY thorough debugging, auditing and metrics.
(Which apparently don't exist.)
--
As for a Heartbleed-esque catastrophe, apparently one is going to be announced for Intel ME at the upcoming Blackhat(?) conference in December. I can't wait to hear about it myself.
Many medical devices run Linux as a User-Interface... (or Windows for that matter).
The actual safety-critical portion of these systems is rarely running Linux, but rather on a bare-metal micro.
I'm reminded of a UAV doing the same thing. It ran L4 for low-level control, realtime scheduling, and security, and then virtualized Linux on top of that.
Sounds unbelievably clunky on the surface, then you realize it's a remarkably useful way to abstract everything cleanly.
Each node does not handle errors at all, but PANICs on a fault. It is up to the supervisor (with global knowledge and state) to handle the fault appropriately.
You can have processes that reboot once every few minutes running for years because people didn't realize they were bugged.
Light on details, but the vulnerabilities are disclosed and fixed [0]. ME updates are already available from many OEMs.
[0] https://security-center.intel.com/advisory.aspx?intelid=INTE...
It is? I learned that in those areas you use different, much simpler and therefore more stable operating systems.
Seems like so many in security fail to see the DoS implication.
There is no solution to bugs other than fixing them. And that's what Torvalds and others have been saying: for a security researcher, finding the bug is the end of the job. For developers, that's just the start.
Linux is not a kernel for this kind of uses. Whoever does it, is doing a disservice to the people.
Operating systems like INTEGRITY RTOS or similar, are the only ones able to match the security quality requirements for such deployments.
It shouldn't be. It's ill suited for that.
Look at seL4, minix3, echronos instead.
https://security-center.intel.com/advisory.aspx?intelid=INTE...
PANIC on fault is exactly what you design into the systems.
What you find is that the truly safety-critical portion of the system is running on a microcontroller and the UI (which is not safety-related) can run on Windows or Linux.
The idea is known as supervision trees.
If the system can continue running, it should do.
You don't want your machine to start crashing after installing the latest kernel. Or at least, that's the golden rule of Linux development.
If phones start crashing after installing the latest Android update, people won't see this as a security/stability improvement. They'll simply see the new version as buggy and of poor quality.
It should have been made a sysctl toggle.
So if that assertion is in the file system, maybe your kernel should die before it corrupts your data permanently.
I too would like to live in a world where sanity preserving assertions have rational consequences. But that world is not this world, and pretending it is won't help you get there.
A deliberate panic could be the basis of a denial-of-service exploit.