Unikernels are secure
unikernel.org
unikernel.org
"There is one mortal sin in computer security (or by the way, in any kind of security) that is feeling safe. It’s just a variation of pride to be true, but it’s very deadly. Blindly trusting a protection technology is an extreme error."
Read up on DEP. Randomizing function addresses at each build is useless when it comes to attack mitigation. You need to randomize addresses at each execution.
ps: BSD have started to randomize at boot time, fun times
Not entirely sure what you're meaning to ask, but yes, Linux does KASLR and tries very hard not to leak pointer addresses from the running kernel to unprivileged userspace:
https://lwn.net/Articles/569635/
It's been enabled by default since the upstream 4.12 kernel, and usually well before that in distros.
(edit I presume that's what the poster above mentioned re: per exec)
BSD also does the same for userspace, but their new feature is that they randomise the kernel per-boot (by re-linking the objects). This is a far stronger form of kASLR (though I think they're calling it something else because it requires relinking the kernel binary).
Even Windows does this well.
Bragging about boot-time ASLR seems quite odd for an article like this, given that traditional operating systems have had similar (and in many ways superior) features like this for more than a decade (PaX introduced ASLR in Linux in 2001, OpenBSD had it by-default in 2003, and Linux shipped it by-default in 2005).
Imagine a standard webapp deployed via CI. Each time any feature is shipped the entire layout is randomized and then deployed. While a redundant system will have multiple copies with an identical layout, that layout will rotate. If required, a CI process could rebuild the system into N parallel groups with N different layouts (your degree of paranoia is the salt to taste here).
Given that these systems are frequently rebuilt with each deploy it seems nearly identical to me in practice to a relink-on-run.
In regards to what could be rebuilt, computer security is plagued with "coulds". What matters is what is done.
I will not play this "Use OpenBSD because it has a new technique that is fashionable" game that I know is being played.
Custom baremetal custon built VMs is not really in the business of distribution appliance images. That's Docker. Different use case. So what IS done is a workflow to rebuild these images when they change.
And that's often. Their HTTP routing table is literally baked into the image. They're not "reusable" for the most part.
Where it fails is in distros. AFAIK most linux distros do something similar (based on Grsecurity patches) but it's useless because... they then distribute the binary, with everything necessary to get addresses.
But if you're distributing it internally/ it's a secret, yeah, you're going to benefit from that.
This is a corollary to "If it isn't tested it is broken". (Another corollary is "Even if it is tested, it may still be broken.") This is more so with security.
Just looking at some of the assumptions, e.g., that the lack of a shell significantly increases the difficulty of an attack, leaves me with doubts about the claims.
I was doubtful about mentioning the lack of shell. Technically it doesn't really do much in terms of security, but it does make a compromise harder. And currently, where we stand, a lot of the security comes from things being really, really hard to compromise. Guessing adresses, not having a shell, potentially no filesystem, etc.
So you do have a noble goal. When I am asked to assess software, to gain a first order approximation, I often look at the culture of the organization that produced it. On the high end, that often translates to some sort of formal process involving un-fun letters like 'ISO' and 'SOC'.
For smaller organizations, or OSS software, I try to get a feel for how the developers seem to be thinking about security. Some of the cultural things I see in your writeup are encouraging--Smaller Codebase, Removing hardware emulation, Cutting off access to ring 0--all bode well for how you are thinking about producing secure software.
But there are some things that are missing from your description. What is your development process? In particular how and when do you attack your assumptions about how the code actually works? What flaws have you found in your testing? Zero findings could mean 1) it wasn't tested 2) it wasn't tested very hard 3) the testers were not very good at security testing.
And as OpenBSD's history has illustrated, some things are totally secure until they aren't.
Precisely. On another domain: "My crypto algorithm has never been broken, it is secure". No it is not, it's just too irrelevant for anyone to be interested in auditing it.
This of course left the host still needing to be secured using traditional approaches but that surface area was smaller.
I'd buy that you can't list that in an article about why all unikernels are already secure, but that seemed to be the heart of the pitch to me when I first started hearing it.
While I certainly sympathize with the idea that the typical kitchen-sink deployment of a full suite of administrative/troubleshooting/build tools on an application server is a security risk, it seems the actual improvements enumerated here are relatively small potatoes.
Apologies for not knowing a lot about the state of the art here, but I'm curious about how these unikernels address VM-internal security (eg, network, filesystem, inter-process level stuff). Do you farm all of that out to the hypervisor? Design around the lack of it? _Is_ there even "inter-process stuff"?
The build should not be relied on to be secure either; certain classes of memory disclosure bugs could leak out enough memory to let you work out e.g. a ROP payload, which wouldn't require injecting any machine code necessarily. Plus, a long-lived machine, or a single instance deployed across many machines, would present insufficient diversity to protect against a determined attacker. The gist here appears to be the notion that their build diversity gives them most of their security, while they've eliminated most of the traditional defense layers (ASLR, DEP/NX, privilege separation, API isolation, etc.)
If the attacker gets her machine code running in the unikernel environment, it's game over no matter what. The machine code can just directly implement a subset of needed functionality without having to call out to unikernel functions.
In short - this post has a hell of a click bait title but really insufficient evidence to suggest that these are truly "secure".
From the original pull request drafting this post:
> While I agree that making hyperbolical claims isn't typically good I'm not sure if it is bad here. There seem to be a tradition in unikernel land with outlandish claims in various blogposts ("Unikernels will kill containers in five years", "Unikernels are unfit for production", etc) so I wrote it with that tone in mind.
But they should definitely add ASLR.
True - I suppose I'm trying to hold onto the "kernel:hypervisor :: process:unikernel" analogy here for no good reason. Following it suggested that it might be the hypervisor's responsibility.
From the article: 'However, if the hypervisor can set up the VM before it is booted and provide paravirtualized interfaces to the hardware, we don’t need access to ring 0.'
Which would be pretty sweet, no?
So say it how it is: unikernels COULD be secure if all this work that needs to be done and we haven't done was done...maybe
The trouble is that instead of uploading "/bin/sh" you'd have to upload the whole shell which you'd want to run; instead of making a syscall for something that a normal kernel would do but this one doesn't, you'd have to compile and upload appropriate code (including device drivers) to get that done.
It'd be a "bring your own machine code" party.
Meaning that if you have a buffer overflow or similar, all you could access was what was in the VM (unless you also happen to pack a VM escape).
That said, i can't help think "DOS in a can" whenever i read about unikernels.
IncludeOS actually came out of a research need to stress-test hypervisors and run 10000 VMs on a single host. Since it is a clean slate system it has some different characteristics and those can be exploited in certain use cases and so people have kept on working on it.
I think - I'm still learning about this embedded stuff.
In general, you could have a unikernel where it's impossible to get executable buffer overflow because there are no memory pages that are both writable and executable - as the article describes, that'd need some help from the hypervisor for the initialization.
Return oriented programming exploits could work anyway, though.
Yes
> whereas a normal operating system would detect the out-of-bounds memory access and kill the process?
No. A buffer overflow in a normal operating system usually results in the the attacker gaining control over the process that the buffer overflow occurred in. This is not as bad as an attacker gaining full system access, but is generally plenty for an attacker to accomplish their goals.
Ring 3 is userspace, you can't interact with hardware or the operating system or anything not in Ring 3 directly.
Ring 0 is everything. There are no restrictions and nothing stops you from writing "Ahahah You didn't say the magic word!" over your entire memory until the CPU crashes.
Having root on a linux kernel is heavily restricted compared to this and still runs in Ring 3 like all other userspace code.
As root, you still have to run the kernel. As Ring 0, you can replace the kernel. Or run your own OS.
In the case of a unikernel deployed on a hypervisor this is not the case, since there is not much else in ring 0 that you wouldn't already have access to from ring 3. Conceptually you can think of the hypervisor as "kernel space" and anything inside the unikernel as "userspace".
There are advantages to running the unikernel solely in ring 3 (eg. immutable page tables) however this is not a requirement for security.
When Ring 0 is compromised, there is no alert or anything to protect the app from compromise. If there is an exploit, it's game over.
However, in Ring 3 and a normal kernel, you get various protections that allow the kernel to recognize some attacks and shutdown the application immediately or even shutdown the kernel.
This prevents a compromised app from running to some extend.
A unikernel cannot do this. If the app is compromised and I don't notice and don't restart it...
Even worse, the attacker could use it as leverage to infect other unikernel based instances of the app to gain some permanence against restarts by simply reinfecting when an instance goes down.
The unikernel is not userspace, not even conceptually. The hpyervisor will not shutdown the app unless it executes illegal instructions. The kernel will shutdown misbehaving programs more easily.
I disagree. There's no reason such mitigations (not sure what exactly you're referring to) can't be implemented by the monitor process (ukvm in the Solo5/ukvm model).
I'd also argue that a normal kernel does not do any integrity checks on the code running in a user process, so the model is exactly the same.
> Even worse, the attacker could use it as leverage to infect other unikernel based instances of the app to gain some permanence against restarts by simply reinfecting when an instance goes down.
For that they'd need to break out of the virtual machine and into the hypervisor / monitor. Which is by no means impossible, but with careful design of unikernel-specific monitors can be much reduced. Of course, I'm by no means suggesting you should back your unikernels with a monitor along the lines of QEMU :-)
2) No, the attacker is most likely there because of some bug in the app, once in the network, it becomes harder to stop the attacker infecting other instances.
3) Hypervisors are not perfect. There are known instances of people infecting the host through the hypervisor.
2) The same applies to any application, not just unikernels.
3) I completely agree.
2) Yes but Unikernels do not provide special protection against this either.
Minor point, but this seems to be a bit lost in the discussion: Generally 1 unikernel == 1 VM (or, virtualization-backed sandbox, the use of "virtual machine" brings too much baggage with it) == 1 application.
So, the attack scope for the class of attacks we're debating is equally limited to a single application, just like on a normal operating system.
When you write an exploit for a normal operating system application, you can't, for example, just write your payload into data memory and start executing it. You can't jump to the address of an array and have the CPU execute it's contents.
On a unikernel this sort of thing becomes trivial since everything is Ring 0 and all protections can be trivially disabled.
You can just write your payload into any arbitrary data field and your exploit only needs to jump to it, even with address randomization this can be exploited (ASLR and similar techniques do not prevent exploits, only make them harder)
The exploiting just becomes a whole lot easier.
It's not even remotely more secure than Ring 3 code running on a kernel that has strict capability enforcement.
If the hypervisor sets the relevant VMCS control fields for the unikernel to cause a VMEXIT on (for example) a load of CR3 and any EPT violations and sets up the guest with a static set of read-only and executable (or in fact, with EPT you could even do execute-only) pages then there is no way the unikernel can disable these protections.
Having said that, I'm not arguing that running unikernels in Ring 0 is the best approach for security, just that it's not impossibly insecure.
With ukvm we're also looking into running all of the unikernel in (virtualized) Ring 3 with nothing running in Ring 0. However, this needs more experimentation since with x86 you can't (for example) easily handle legitimate exceptions (e.g. divide by zero) from Ring 3.
(VM can even run VM as a guest, recursively, which is useful for developing the newer version of VM on a machine other people are using for other work.)
Therefore, we can evaluate these claims by asking what we know about security breaches on VM/CMS systems.
There is what amounts to an "API". The virtualized software thinks it's talking directly to hardware, so the API is just an emulation of that hardware. Since controlling hardware is almost invariably messier than doing system calls, the API is actually more complex.
Therefore, we can evaluate these claims by asking what we know about security breaches on VM/CMS systems.
Here's an HN post I wrote a few years ago, about VM exploits: https://news.ycombinator.com/item?id=9241807
Plenty of exploits to be found, many of which were related to emulation of "I/O channel programs", which are, essentially, System/370 I/O "hardware".
Nobody said a hypervisor was completely trivial, but just to be clear, a pure hypervisor wouldn't "emulate" any hardware except what was physically present on the system. The line between "virtual machine" and "hypervisor" and "emulator" are blurry enough as it is, and I'd like to be clear about what the subject of this little thread is.
> Here's an HN post I wrote a few years ago, about VM exploits: https://news.ycombinator.com/item?id=9241807
> Plenty of exploits to be found, many of which were related to emulation of "I/O channel programs", which are, essentially, System/370 I/O "hardware".
Very interesting. Thank you.
Timmy the software developer and his many years of forensic analysis have led him to believe that you need a shell to root something. Literally the entire page's takeaway about security is "It's hard to use, so it's secure!"
But the whole point of userspace is to provide privilege separation between the kernel and userspace. In unikernels everything is in the kernel, and you can freely mutate in-kernel state without any segmentation violations. In other words, of course there are no syscalls when there is no separation between user code and the kernel ("no syscalls" is the description of a unikernel). So, what is the tangible security benefit?
What I want to know is how do you protect against in-kernel ROP? Or hell, just plain old-fashioned stack overflows? The ASLR mentioned is not really useful because it's only on-recompile (Linux's ASLR is on-execution). The only experience I've had with writing shell code was messing around with https://microcorruption.com/ for a few evenings, and even I know that putting everything into the same address space is just asking for trouble (of course you can mitigate it, but praising the premise as a feat of security seems to be missing the point to me).
Maybe I'm just massively misinformed, but I simply don't see how someone can proclaim that "unikernels are secure" by just re-stating the premise as the justification for their security (and then following up with a bunch of hypotheticals).
You could potentially compromise the rump kernel, but you still wouldn't be able to break out of the VM's isolation context.
On top of that, you didn't get to reap the benefits of any of the protections you get from a proper Kernel like Ring 3 execution, NX bits on pages, guard pages on the stack, etc.
You can prove that it's secure in that the algorithm itself does not leak information.
(As an example of my understanding, rot13 does nothing to randomize distribution of characters, so it "leaks" information about what it has modified. This sort of leakage can be proven as absent from your algorithm. Anything else is a bit tougher.)
In other words, those proofs don't have as much real-world significance as you'd like.
This argument is completely incoherent.
The author has presumably confused the concept of a POC, which is an exploit reduced and simplified for the consumption and understanding of laypeople, with that of a real exploit.
A unikernel application which only executes native code[2], combined with a hypervisor / hardware that enforces immutable page tables and NX, AFAICT leaves ROP/JOP as the sole remaining attack vector for RCE. Further, each deployment of the application should be done via a new build (or at least re-link), which can employ the build-time ASLR Per mentions to mitigate ROP/JOP attacks.
I'd be interested in your thoughts on what other relevant attack vectors I've not thought of that would allow for a persistent compromise of a unikernel. Feel free to discuss here or on the devel.unikernel.org forum thread[3].
[1] Page 4 in this whitepaper from 2015: https://wiki.xenproject.org/images/3/34/XenProject_Unikernel...
[2] Does not contain a JIT or interpreter able to "eval()" new code.
[3] https://devel.unikernel.org/t/unikernels-are-secure-here-is-...
Only executing native code is no defense at all. You're not even stopping attackers from the '90s like that.
ROP is only a hardship until exploit code can find a way to allocate executable memory or hook the program's own dynamic execution (note: this is not necessarily a "JIT" or "eval") capabilities. ASLR only matters when exploits can't find infoleaks.
Really, what you're describing here is the browser security model. When staffed by the largest, best security teams in the world, the browser security model almost works.
I'm not familiar with these. Googling "MOSDEF attack" suggests Massive Attack songs and "CORE IMPACT" various products by Core Security, I presume you're referring to neither? :-)
> Only executing native code is no defense at all. You're not even stopping attackers from the '90s like that.
> ROP is only a hardship until exploit code can find a way to allocate executable memory or hook the program's own dynamic execution (...)
Only executing native code loaded at deployment time, i.e. presumably from a trusted source. What I meant by "immutable page tables" is that after loading the unikernel image the set of executable pages is fixed, and all of those are read only. This is not something current unikernels do, but that's more due to lack of development time rather than any inherent implementation complexity.
If there is no way for the exploit code to ask the hypervisor for new executable pages or overwrite existing ones then the amount of damage it can do and its ability to persist in the running system is greatly reduced.
> Really, what you're describing here is the browser security model. When staffed by the largest, best security teams in the world, the browser security model almost works.
It's similar but not the same. The browser security model breaks down in a large part due to exposing way too many APIs, most of which are too complex to sensibly audit. Linux system calls have a similar problem, which is why you have to hand-craft things like seccomp profiles to match your particular application.
Unikernels (as used to implement network services) lend themselves extremely well to running on top of a well defined API / sandbox boundary, designed with security in mind. Again, we're not there yet, but the steps we need to explore in this direction are fairly clear.
#include <linux/seccomp.h>
#include <linux/filter.h>
#include <linux/audit.h>
#include <sys/prctl.h>
#include <asm/unistd.h>
#include <stddef.h>
prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
struct sock_filter filter[] = {
BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, arch)),
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, AUDIT_ARCH_X86_64, 1, 0),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL),
BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, nr)),
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_fork, 4, 0),
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_vfork, 3, 0),
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_clone, 2, 0),
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_execve, 1, 0),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL),
};
struct sock_fprog fprog = {
.len = sizeof(filter) / sizeof(filter[0]),
.filter = filter,
};
prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &fprog, 0, 0);
SystemCallFilter=~fork,vfork,clone,execve
Or in other words, the argument isn't "unikernels are secure," it is "programming in an environment without fork/exec is secure," and unikernels are a needlessly complicated way to provide that environment.
And mmap(), mprotect(), ptrace(), ..., $syscall_du_jour(). It's not about fork/exec, it's about limiting the APIs available to the unikernel by default and by design.
Yes, you can do this for conventional Linux applications. But I wouldn't call it super easy, at least not for the vast majority of developers out there.
Changing all your code to respect this standard is strictly less hard than porting it to a unikernel, because it's one step of porting it to a unikernel.
Yeah, they're direct function calls now. How exactly is that more secure? The author's right that many traditional exploit paths are gone, but only because they've been replaced by even easier ones.
I think kernels try not to leak the address of system calls at run time, and if they are scattered around in a 64bit address space they are tough to search for. Educate me!
Developers love to write code, but unless unikernel developers start speaking up instead of writing code, the major clouds won't get on board and provide the needed fast boot times plus pricing, which makes all this unikernel code not a lot more than interesting personal / academic projects.
Questions of unikernel security are certainly academic if the cloud infrastructure doesn't run them in the most effective manner.
The point being that docker is actually a really clunky thing that is reinventing the virtual server ecosystem within the OS which is pointless.
Serverless systems are better implemented as unikernels than docker images.
Unikernels make it possible to boot an entire OS in milliseconds, service an inbound web request, then disappear.
It shouldn't surprise anyone that feature X is available in C++. But I acknowledge there are tons of shotguns strewn around everywhere in C++ so you can blow your feet off.
The "vector into the VM" might not be as obvious. For example, in a network security contest in college I compromised the source repository of the server to give my team an advantage and won. That was before git it was just a folder on the server. Others argued it wasn't fair but the instructor sided with me since that's a plausible scenario in real life.
There also tempest-like attacks for extracting private keys and such, so it's there at also at least the 3rd vector - from VM to the hardware as well. And forth from VM to the network (but I guess these can be subsumed in the "to the hypervisor" case).
I suppose the hypervisor is more minimal and secure. Tenenbaum, thou art avenged.
I think you could describe that as "more secure" or perhaps more generically "safer" rather than "secure."
What does it mean for something to be secure? As I see it, for something to be secure, it means that every security invariant holds under all attacker models.
Let's break this down. What's a security invariant, and what's an attacker model?
A security invariant is a statement like "information may only be read by principals authorized to do so" or something similar. The blog post talks about "oh well every exploit I've seen pops a shell with /bin/sh and we don't have /bin/sh so you can't hack it." This attacks the symptom and not the disease. If the attacker is in a position to run /bin/sh you are already at a disadvantage, the attacker was previously going to break your security invariant by spawning a shell and then copying out your secrets. They could instead just use their code execution to read the secrets directly, ala heartbleed, if they are in the same address space.
An attacker model describes the attacker, their motivations/goals, and capabilities. For example, what if the attacker wanted to take your unikernel enabled app and repurpose it into an e-mail spam bot? Well, there isn't a writeable FS and there isn't the ability to exec stuff, so the old model of "copy some files up and run them" won't work. However, is there executable memory in general? Can the application allocate existing executable memory? If the application is, or involves, for example a modern JavaScript interpreter, then it has to be able to do this in some capacity. So now the attacker modifies their methods so that their goal can be achieved in the constraints of this new system.
There is an argument you can make about sandboxing and principle of least privilege, but that argument transcends unikernels really. Look at the sandboxing and separation strategies used by programs like chrome, vsftpd, and qmail. They (in conjunction with enforcement technologies like seccomp-bpf or selinux) can make the same guarantees about the lack of system calls and least privilege.
I'm pretty sure that the title of this post really should be "unikernels are obscure" because that's the layer of protection you're really getting here.
One of those things doesn't belong there. Guess which?
:)
>"Unikernels on the other hand don’t have system calls. They only have function calls."
If my program is compiled against glibc functions which wrap actual system calls how does my program work when there is no userland?
Is there a compiler that compiles my program to a unikernel target? Can someone explain how this works?
> * packet interface for the network
> * a block interface for some storage
> * a serial port to output console data
Some applications need additional devices, such as for example hardware RNG, atomic clocks and/or GPGPU.
i.e. Just because I can't tell if you have two hands to type this doesn't mean you don't have them.
"You have two hands if and only if there's a formal proof that you have two hands"