Could you explain how SELinux could ever sandbox against RCE in sshd? Its purpose is to grant login shells to arbitrary users, after all.
But, the normal login or remote command execution is at least audited. And it might have other enforcement of which accounts or programs are permitted. A configuration disallowing root could not be bypassed by the sub-process.
You could also decide to run all user logins/commands under some more confined SE-Linux process context. Then, the actual user sessions would be sandboxed compared to the real local root user. Of course, going too far with this may interfere with the desired use cases for SSH.
At this point “All libraries could be malicious” is a threat model that must be considered for something as security critical as OpenSSH.
Keep in mind that upstream didn't even link to liblzma. Debian patched it to do so. OpenSSH should defend against that too?
This is a Linux problem, and the problem is systemd, which is who brought the lib into memory and init'd it.
Not at all, it is a distro issue because a few distros such as Debian chose to patch openssh to bring in systemd support [1].
Other systemd-based distros like Arch Linux remains unaffected because they don't carry this patch.
1: https://sources.debian.org/src/openssh/1%3A9.7p1-2/debian/pa...
What a complete failure of an init system's job, and the patch was applied due to systemd not resolving the issue in another way.
This is the problem with systemd. Way, way way too much complexity.
It's not a systemd issue, it's a larger cultural issue with packagers increasing attack surface to make their lives easier.
Both Redhat and Debian and others applied this patch as a result. People didn't do it "just cause".
https://bugzilla.redhat.com/show_bug.cgi?id=1381997
Jakub Jelen 2016-10-12 08:40:44 UTC
<snip>
> Why do you want to avoid sd_notify()?
Why to avoid this? Well, it is adding more systemd-specific bits and new build dependency to something that always worked well under other inits without any problems for years.
https://news.ycombinator.com/item?id=39878181
So no, as Pottering claimed, sshd would not be hit by this bug except for this systemd integration.
I really don't care about "Oh, someone could have written another compromise!". What allowed for this compromise, was a direct inability for systemd to reliable do its job as an init system, necessitating a patch.
And Redhat, Fedora, Debian, Ubuntu, and endless other distros took this route, because something was required, and here we are. Something that would not be required if systemd could actually perform its job as an init system without endless work arounds.
Also see my other reply in this thread, re Redhat's patch.
After all, what you're saying is and has always been the case! It's like saying "Well, Ford had a design flaw in this Pinto, and sure 20 people died, but... like, cars have design flaws from time to time, so an accident like this would've happened eventually anyhow! Oh well!"
It doesn't jive in this context.
Directly speaking to this point, patched ssh was chosen for a reason. It was the lowest hanging fruit, with the greatest reward. Your speculation about other targets isn't unwarranted, but at the same time, entirely unvalidated.
That said, there are dozens of ways to fix this and it really seems like RedHat chose the worst one. They could have patched sshd in the other various ways listed in that ticket, or even just patch it to exit on SIGHUP and let systemd re-launch it.
(Some difficulty with this one though. For instance you probably have to ban running arbitrary code at load time, but you should do this anyway because it will stop people from writing C++.)
Darwin doesn't let you make library regions writable after dyld is finished with them. (Especially iOS where codesigning also prevents almost all other ways to get around this.)
Something like OpenBSD pledge() can also revoke access to it in general.
> But x86-64 got rid of the middle rings.
x86 is a particularly insecure architecture but there's no need for things to be that way. That's why I mentioned PAC, which prevents other processes (including the kernel) from forging pointers even if they can write to another process's memory.
So maybe just don't patch sshd?
sshd is not the problem. the ldd/monolith architecture surrounding systemd is.
What if I duplicated this attack but instead targeted dbus or any other thing that systemd is managing?
The fact, that the whole reason this library is even being pulled into the sshd daemon process, is some stupid stuff like readiness notification, which itself is utterly broken on systemd, by design (and thus is forever unfixable), and makes this even more tragic.
Don't put your head into the sand, just because of the controversial nature of the topic. Systemd was VERY accommodating in this whole fiasco.
Saddest part of all this is, that we know how to to do better. At least since Bernstein, OpenBSD and supervision community (runit/s6) guys solved it. Yet somehow we see same mistakes repeated again and again.
If you want to read about notification shenanigans and systemd's dubious (at best) decisions and implementation quality, read here: https://jdebp.uk/FGA/unix-daemon-readiness-protocol-problems...
I believe this is how you do readiness notification properly: http://skarnet.org/software/s6/notifywhenup.html
I.e. you fork and run little helper to write, or directly write a single byte(!), to notify supervisor over supervisor provided fd. It allows you to even privseparate your notifier stuff or do all the cute SELinux magic you need.
But that would be too simple, I guess, so instead we link like 10 completely unrelated libraries into sshd, liblzma being one of them, one of the most crucial processes on the machine. To notify supervisor that it's ready. Sounds about right, linux distros (and very specific ones at that).
Sshd should be sacred, nothing more than libc and some base cryptolibs (I don't remember whether it still needs <any>ssl even) it needs.
Another great spot to break sshd is PAM, which has no place doing there either. Unfortunately it's hard dep. on most linux distros.
Maybe sshd should adopt kernel taint approach: as soon as any weird libraries (ie everything not libc and cryptolibs) are detected in sshd proces it should consider itself tainted. Maybe even seppuku itself.
The exploit could be, probably, somehow doable without systemd. But it would be much, much harder though.
Don't try to obfuscate that very fact from the discussion.
> Another great spot to break sshd is PAM, which has no place doing there either. Unfortunately it's hard dep. on most linux distros.
There are many things to hate about PAM (it should clearly be a system daemon with all of the modules running out of process), but there's literally no universe where you get to claim that sshd should have nothing to do with PAM - unless you want to plug every single possible authentication mechanism into sshd upstream you're going to end up with something functionally identical.
Not true. They have this ability only for binaries that are linked to liblzma. If sshd were to be decomposed into multiple processes, not all of them would (hopefully) depend on all the libraries that the original sshd depended on.
Exactly how to do it (especially transparently for the user), I have no idea though. Maybe sending ssh login requests to two different sshd implementations and if they don’t do the same things (same system calls), they are both killed.
Or some kind of two step login process where the first login only gives access to the sandbox of the second login process.
But in general I assume the Linux attack surface is too big to do software diversity for all of it.
RCE doesn't really follow a login process design. As soon as you got RCE you can be considered pwned.
If not now, then at the time the next locally exploitable vulnerability comes up. There are plenty.
by default sshd has access to all files in /home/$user/.ssh/, but that could be prevented by giving private keys a new unique file context, etc.
SELinux would not prevent all attacks, but it can mitigate quite a few as part of a larger security posture
> Libselinux pulls in liblzma too
libselinux is just an unwitting vector to link liblzma with openssh
This is sort of overlooking the problem. While true, the processes spawned by sshd do need to be able to do all these things and so even if you did sandbox it, preserving functionality would all but guarantee an escape is trivial (...just spawn bash?).
Now, sandboxing sudo (in the general case) with SELinux probably isn't possible.
sshd is a sitting duck. Bifurcating sshd into a multimodule scheme won't work because some part of it still has to be loaded by systemd.
This is a web of trust issue. In the .NET world where refection attacks happen to commercial software that features dynload assemblies, the only solution they could come up with is to sign all the things, then box up anything that doesn't have a signing mechanism and then sign that, even signing plain old zip files.
Some day we will all have to have keys, and to keep the anon people from leaving they can get an anon key, but anons with keys will never get on the chain where the big distros would ever trust their commits until someone who forked over their passport and photos got a trustable key to sign off on the commits, so that the distro builders can then greenlight pulling it in.
Then I guess to keep the anons hopeful that they are still in the SDLC somewhere their commits can go into the completely untrusted-unstable-crazytown release that no instutution in their right mind would ever lay down in production.
[edit: sibling sweetjuly said it better than I could. I doubt that this much more than a fig leaf on any real world system given what sshd is required to have to do.]
Which is nice and all, but that implies a "parent" system that creates and deploys those systems. Which people likely want remote access to.. Probably by sshd...
You can limit the exposure of the system from RCE in sshd with SELinux without preventing legitimate users from administering the system.
Granted that SELinux is overly complicated and has some questionable design decisions from a usability standpoint but it's not as limited or inflexible as many seem to think.
It really can stop a system service running as "root" from doing things a real administrator doesn't want it to do. You can couple it with other mechanisms to achieve defense in depth. While any system is only as strong as its weakest link, you can use SELinux to harden sshd so even with exploits in the wild it's not the weakest link vis-a-vis an attacker getting full unconfined root access. This may or may not be worth your time depending on what that box is doing and how connected to the rest of your infrastructure it is.
There seems to be a pervasive misunderstanding of the difference between standard UNIX/Linux discretionary access control and SELinux-style mandatory access control. The latter cannot be fooled into acting as a confused deputy anywhere near as easily as the former. The quality of the SELinux policy on a particular system plays a big part in how effective it is in practice but a good policy will be far harder to circumvent than anything the conventional permissions model is capable of.
Moreover, while immutability is obviously an even stronger level of protection, it is not necessary to make the system immutable to accomplish what I've described here while still allowing legitimately and separately authenticated users to fully administer the system.
The amount of people confusing DAC and MAC is concerning. You’ve done an excellent job explaining the topic.
Most people turn SELinux off anyway, so they have no clue how it operates.
DACs (discretionary, unix perms) are DACs and MACs (mandatory, SELinux) are MACs. They are mandatory - it's in their name.
Think of SELinux as completely orthogonal access control system, that can overturn any DAC decision, which it in fact does. SELinux language is much more featured than DAC language, it can express domain transitions.
Nobody here has inspected the sshd_t policies but I believe exec transition should be forbidden for arbitrary binaries (I hope).
That should in essence thwart arbitrary exec from remote key payload. If actual shellcode would be sent though (e.g. doing filesystem open/write/close), that is a little bit different.
This backdoor does not bypass remote authentication so it should be able to transition to the new domain that has access to these files
But in the more common case it just doesn't: you have an sshd running on a dedicated port for the sole purpose of running some service or another under a specific sandboxed UID. That's basically the github business model, for example.
Maybe write a shell to coordinate between them? It should support piping and output redirection, please.
I'm saying, as a response to the point above, that sandboxing sshd is absolutely a valid defense-in-depth technique for privilege isolation, that it would work against attacks like this one to prevent whole-system exploitation, and that it's very commonly deployed in practice (c.f. running a git/ssh server a-la github).
We'd need something more like a cryptographically attested setreuid() and execve() combination that would run only commands signed with the private key of the intended user. You'd want to use a shared clock or something to protect against replay attacks
Whether it's worth it or not depends on circumstances. In many cloud environments, nuking the VM instance and starting over is probably easier than fiddling with SELinux.
at minimum, ipv6 only if you absolutely must do it (it absolutely cuts the scans way down)
better is to only host it on vpn
even better is to only activate it with a portknocker, over vpn
even better-better is to set up a private ipv6 peer-to-peer cloud and socat/relay to the private ipv6 network (yggdrasil comes to mind, but there's other solutions to darknet)
your sshd you need for server maintenance/scp/git/rsync should never be hosted on ipv4 clearnet where a chinese bot will find it 3 secs after the route is established after boot.
It's not like this is SSH's fault anyway, a supply chain attack could just as well backdoor some Fortinet appliance.
I'd stop at SSH, since I don't see a conceptual difference to how a VPN handles security (unless you also need to internally expose other ports).
The only software remotely in the same league is a stripped down Wireguard. There is a reason the attacker decided to attack liblzma instead of OpenSSH.
If you have password authentication disabled then it shouldn't matter how many thousands of times a day people are scanning and probing sshd. Port knockers, fail2ban, and things of that nature are just security by obscurity that don't materially increase your security posture. If sshd is written correctly and securely it doesn't matter if people are trying to probe your system, if it's not written correctly and securely you're SOL no matter what.
I fail to see a problem here.