Searching for the killer app of unikernels
unikernel.org
unikernel.org
Unikernels are a great technology and very much ready for prime time, but unless cloud vendors provide the millisecond boot/teardown/pricing, Unikernels will remain nothing more than a niche special interest technology.
All the technology is in place for Unikernels to take off once the cloud vendors get behind them.
I wrote Bootrino specifically for the purpose of booting Unikernels (and a whole host of other exotic tiny operating systems) on the major clouds and it works fine - you can see a demo here: https://www.youtube.com/watch?v=zJhxmVlR46c (Bootrino not released to public as yet).
However, the real draw for Unikernels is short lifetime function as a service type applications.
For what it's worth, though, there's a lot of value in long-running unikernels. There are good reasons not to pay even the cost of a memcpy of the unikernel image, especially when it comes to making good use of system resources. I'm mostly interested in unikernels on bare metal (or something close, like device-specific PCIe passthrough). There's a lot of potential in an application having full and exclusive access to a hardware platform; it means that your application can make creative use of isolation primitives, be aware of its true resource limits, and choose scheduling and work sharing primitives which make sense for the application.
In many ways, good unikernel environments are similar to DOS, I'm amused to see the resurgence.
Solo5 acts as the host for MirageOS and maybe other Unikernels, or even just for raw C++ code.
So how is this better than using existing IPC mechanisms and using the "user" abstraction to separate privileges?
And how is this more secure then IPC to a process running under another user given
> The browser takes a careful look at the pointer and the data it is pointing at. This code will need to be properly vetted as a buffer overflow here could potentially allow the unikernel to execute code outside its hardware prison.
(The long list of CVEs tells us how good programmers are at "properly vetting" buffers).
It's a smaller, cleaner, interface that's been designed from day 1 to be secure, rather than having security awkwardly partially retrofitted.
> (The long list of CVEs tells us how good programmers are at "properly vetting" buffers).
Right, which is exactly why it's vital to keep those interfaces as small as possible, rather than having dozens of different IPC mechanisms for vulnerabilities to hide in.
Care to argue how something based on sharing memory could ever be more secure than sending messages over pipes or sockets (between different processes)?
* For example, DoS-safe
... where the sharing is mediated by the kernel. You can't access the pipe/socket buffers directly from a user-space program.
EDIT: IIRC, POSIX message queues, at least on Solaris, are implemented SHM and clever queue bookkeeping. But MQs are not sockets or pipes.
There's no fundamental reason shared memory should be less safe than sockets (say); ultimately it's just bytes. Shared memory has a bad reputation because of how it behaves when processes sharing memory run concurrently, especially if they're both writing, but that's not a concern when we're talking about a unikernel interacting with the host, because there's no need to have anything happen concurrently there.
So you choose ONE for the message-passing mechanism. If the process does not incorporate code that opens a pipe and reads from it, you cannot force it to do so. Alternatively, there are all of these and you want to add VMCALL to that soup as well.
> There's no fundamental reason shared memory should be less safe than sockets (say); ultimately it's just bytes.
We're talking about a unikernel writing to the hypervisor's address space. A bug in the hypervisor's validation (or another exploit) will allow it to overwrite arbitrary memory in the hypervisor. Sockets will not.
But then you find that somewhere in the lower layers of the framework/interpreter you were using there's something that relies on it. You disable all the sysctls you thought you weren't using and then your image library turns out to need them. Once the functionality has ever existed in the ecosystem, it's very hard to get away from.
> Alternatively, there are all of these and you want to add VMCALL to that soup as well.
I want to deprecate all the others at the same time, otherwise there would indeed be no point.
> We're talking about a unikernel writing to the hypervisor's address space. A bug in the hypervisor's validation (or another exploit) will allow it to overwrite arbitrary memory in the hypervisor. Sockets will not.
What's the difference between a unikernel sending data to the hypervisor and a process sending data to the kernel? The data always has to cross the boundary one way or another. Why would one approach be any more or less safe than the other?
So you refrain from using the framework in question because the very same framework wouldn't work at all in the hypothetical VM because files do not exist there. You have to write everything from scratch using only your IPC mechanism of choice, just like with VMCALL.
> What's the difference between a unikernel sending data to the hypervisor and a process sending data to the kernel?
Ultimately, trust.
1. The developers. I have more confidence in the implementors of, say, Linux kernel, than in developers of, say, Chrome or Firefox or Safari.
2. Testing. Kernels routinely handle successfully vastly more corner-cases than web-browsers. [Yes, I see upon all of current Linux, Windows, etc. installations as continuous, live, in-production testing.]
If kernels were as shabby as today's browsers, nobody would use them for anything but toys. On the other hand, try to find a person who gets genuinely surprised when a web browser crashes.
It's always going to take a new standard and a new community/ecosystem that fits things into that new standard or rewrites the things that can't be fitted. The only way such an effort can ever hope to succeed is a) an extremely clear standard with a rigorous conformance test b) a way to run legacy programs that don't comply with the standard c) but in such a way that there are still incentives for programs to move onto the new way.
In theory we could create a stripped-down linux that only offered one limited form of IPC and then start porting libraries and processes to run on that stripped-down linux. But it would be difficult to get a consensus on exactly what system calls etc. are allowed, and what would you do for running legacy processes? You have to add the alternative IPC systems etc. back in and add an opt-out mechanism, and then a) every component, even well-behaved ones, loses the advantages of the new system (fast booting, low overhead and so on) b) there's very little incentive for a component to ever be ported to the new system.
> 1. The developers. I have more confidence in the implementors of, say, Linux kernel, than in developers of, say, Chrome or Firefox or Safari.
I'm not sure where web browsers comes into it. I have a lot more confidence in the unikernel folks than the container folks from what I've seen of their tool choices and code quality.
> Kernels routinely handle successfully vastly more corner-cases than web-browsers.
Kernels get very little direct exposure to actively hostile userspace programs, and local privilege escalation vulnerabilities are relatively common - indeed I'd say more common than browser vulnerabilities.
VMCALL is too flexible for being able to call it a "standard". What does the original article suggest? Yeah, define a struct and pass a pointer to it. As "standard" as defining protobuf messages.
Heck, even the "big" (Xen, HyperV, VMWare, VirtualBox) hypervisors don't agree on their hypercalls.
The problem of having multiple incompatible OS-interfaces is repeating itself, just one level below, on the hypervisor level. Madness.
Ironically, these problems would not exist had the world gone the way of IDL and microkernels.
> b) a way to run legacy programs that don't comply with the standard
No-go unless the VM implements all of the stuff that you want to get rid of.
Plus, whatever you decide to run in a VM may want to talk to a service in another VM. How do they talk to each other? [continue the exercise until you reimplement one or more of existing IPC mechanisms].
I see it as another path to a very similar end state. A unikernel running under a hypervisor is much like a process running under a microkernel.
> No-go unless the VM implements all of the stuff that you want to get rid of.
Legacy components can run under a traditional linux VM running under the same hypervisor. If you've got, say, a gnarly image processing piece that needs to talk to a task queue over unix domain sockets or something, you run them both in the same VM; you can get on with your business, but at the same time that doesn't impact the isolation of your other components, performance/scalability cost is only borne by the component that's causing it, so there's an adequate incentive to properly isolate that component (or maybe not - if it's not causing reliability or performance issues it can stay as it is, that's fine too).
> Plus, whatever you decide to run in a VM may want to talk to a service in another VM. How do they talk to each other?
In a way that's always traditionally been used for untrusted communications (e.g. network sockets), so that they know not to trust each other.
So your "trusted" components now incorporate a TCP/IP or another network stack. Plus you'll somehow have to authenticate the other end...
And yes, whenever one component talks to another they have to authenticate themselves; that's much of the point.
If the routing unikernel is untrusted, how are you going to prevent another unikernel to impersonate it and route on behalf of it?
> And yes, whenever one component talks to another they have to authenticate themselves; that's much of the point.
So... separate processes running under different users + unix or TCP sockets + user-based firewall rules does the thing?
If an attacker compromises it the worst they can do is block messages causing a DoS.
> So... separate processes running under different users + unix or TCP sockets + user-based firewall rules does the thing?
Yes except that with that approach you still have all the other ways those processes could interfere with each other.
Yes, you're right, add to that seccomp: http://man7.org/linux/man-pages/man2/seccomp.2.html
Some more closely meet the definition of a Unikernel, others are more like stripped down Linux, others are more exotic OS's that are nothing more than a network driver and disk driver in a virtual machine, such as Solo5.
I got Bootrino (see elsewhere) to boot all of these types of operating systems including embedded Linux (Yocto), Solo5, TinyCore, Alpine, and some other weird run-from-RAM OS configurations.
I got MirageOS running on all of the above, so unikernels can fit within other OSs or live on their own.
Also, there's a small set of programs that don't get along with the abstractions provided by OSes and run better on a virtual hardware abstraction instead.
The intrinsic overhead of the OS (i.e. costs of mode switching) is much smaller than you might think.
It's true that some programs don't really want an OS getting in their way, but a modern operating system provide decent facilities for getting _out_ of the way (e.g. O_DIRECT for block I/O, pinning tasks to cores) and that's usually what those things use.
A userspace process on x86 has a certain amount of overhead, namely the overhead needed to switch between it and the host kernel. This takes the form of SYSCALL/SYSRET (mostly) for system calls and interrupts/IRET (mostly) for interrupt returns. SYSCALL and SYSRET are quite fast. interrupts and IRET are slow.
A VM guest needs to do a VM exit (which is quite a bit slower than even an interrupt, I believe) and then fix up a bunch of host state that doesn't get restored by the CPU. Then, to get back to guest mode, it needs to manually restore some guest state and do VMRESUME.
Regular user mode wins hands down.
To be fair, unikernels get to directly create page tables, which is potentially very useful. But, for the simpler applications, it's probably not.
(I more or less maintain the glue between x86 Linux user code and the kernel. I do, in fact, know exactly how this works.)
Anyway, I don't personally think reducing system call overhead is interesting but it seems to be one of the arguments unikernel people make.
One of the justifications: it removes the need to copy data between kernel/user-space.
KTLS strikes me as inelegant. Socket programming requires you to reason about your frames on the wire. Proto-socket features complicate this.
A unikernel approach would allow us to eliminate unnecessary copies, and could be done with more elegance than KTLS.
What would happen if you used special instructions to try to implement a non-cache-polluting memcpy? Or is that even possible? It's been a while but I seem to think that modern CPUs have a variety of cache control instructions. Zero copy is great but it can be a PITA.
The kTLS paper showed pretty small speedups, so maybe the real motivation is TLS hardware offload.
...and with no way to run ancillary scripts and cronjobs e.g. log management, backups, security tools... Nor ssh into the VM to inspect it and run debuggers or monitoring and metric agents.
Essentially you are giving away all the practical advantages of modern OSes. There's a reason why the usual famous tech companies don't use unikernels, generally speaking.
As for memory management, in the Solo5/ukvm case the amount of memory you grant to the guest (monitor process) is essentially the maximum you expect the guest to need, ever. So it's similar in terms of overhead to many usermode allocators which just mmap() a huge block of memory at start and then hand out to the application from that.
FV guests can similarly have a balloon driver, that doesn't mean the interface isn't awkward.
I'm not buying into this myself as of yet but at least in theory this could lead to.an "application provably correct all the way down the stack".
If you compare the attack surface of a Linux user process that is very tightly seccomped to the attack surface of a whole guest VM (with all the awkward emulation needed), it's not at all clear to me that guests are better.
I'm not sure I follow the logic in your harder-to-retrofit opinion. In unikernels the central idea is retrofitting the unikernel around the host interface and the application. It seems obvious to me that this would be more easier to develop and debug inside a standard process, than an emulation of bare iron, even if you would have to call it a different platform due to a radically smaller syscall interface.
I don't see that they have to be? There's no reason for the host to offer e.g. a virtualized graphics card except to unikernels that specifically want to output graphics.
> In unikernels the central idea is retrofitting the unikernel around the host interface and the application.
I don't think I follow? The only part that's retrofitted is using an interface that was designed to host OSes to instead host an application, but it's a good fit because virtualized guests haven't been treated as trusted (particularly not in contexts like AWS), whereas for decades of unix history any local process was treated as more-or-less trusted.
> It seems obvious to me that this would be more easier to develop and debug inside a standard process, than an emulation of bare iron, even if you would have to call it a different platform due to a radically smaller syscall interface.
If we're talking about creating a new platform with a restrictive, strictly secured syscall interface, whether we talk about a hypervisor running unikernels or a kernel running processes is pretty much just terminology. I would think the actual interface specification would end up very similar to the current hypervisor/unikernel one.
The big advantage of keeping compatible with existing VMs is to provide a way to run legacy processes without compromising the security of new-style ones (and while making sure the incentives in terms of e.g. performance run in the right direction). If one component of your system needs to be run as a traditional unix process, that's fine - it gets its own VM in which it's running a traditional kernel and the actual process - but it can't interfere directly with your other components, and it pays for its own overheads.
Historically Unix processes and VM guests have both been treated as untrusted on paper, but with not much scrutiny. There are frequent VM breakout vulnerabilities announced and there's wide agreement that this is a porous security boundary. The numbers are probably better for the VM interface, but it's in the same category.
Apparently we agree that in terms of exposed functionality, the end result is not going to be very different, if we restrict the VM interface vs use seccomp or similar.
A difference here is that there is already an easy and widely supported and understood system for this restriction for the process case. And processes are easier to debug, instrument, monitor and manage. And they don't require platform support for virtualization, which can be useful on eg embedded platforms, phone apps etc.
For the last point about running legacy processes, maybe we have a different understanding of what Unikernels are. If you have an unikernel application that uses a Rump kernel or MirageOS, you are not going to be able to run legacy processes in the Unikernel container in any case, there is not even going to be a kernel/userspace boundary. There is just one code image with everyting linked into it. So the traditional VM interface is not going to help with the legacy processes.
If you were talking about running regular VMs for legacy apps alongside unikernel images, then that's of course valid, but I don't agree that this is an argument for using the VM interface for unikernel images.
There is? I see a bunch of ad-hoc, retrofitted efforts, which seem to be constantly playing whack-a-mole in terms of what they forgot to restrict.
> And processes are easier to debug, instrument, monitor and manage.
Yes and no. The kernel/process boundary can make debugging and profiling much harder when you need to do so across it. With a lot of unikernels you can run the whole thing as a userspace process for development purposes and debug/profile without worrying about that. And these days we have a lot of good tooling for monitoring and managing VMs.
> If you were talking about running regular VMs for legacy apps alongside unikernel images, then that's of course valid, but I don't agree that this is an argument for using the VM interface for unikernel images.
That's what I meant. If you have a mixed deployment of legacy components and new-style components, that approach lets you realise the benefits quicker for the new-style pieces and ensure there's an incentive to move components into the new style. Whereas if you just run a bunch of processes, then no single component can benefit from isolation until you've ported everything.
The debugging would be similar with a unikernel run as a seccomp process: you could debug it as a single process. I believe the BSD rump kernel is already used like this for debugging purpouses.
If you want to mix legacy processes and unikernels in parallel, you would still run the legacy processes on a different VM in the seccomp case.
Why is this better than running as a normal user process? Look at the complexity inherent in securely sandboxing applications using existing interfaces -- the Linux system call interface amounts to hundreds of system calls, with different mutations for different architectures and many calls multiplexing even more operations (eg. ioctl()) the numbers grow even higher.
Unikernels give us a shot at redefining the process interface, while still being able to run on existing infrastructure (host OS or hypervisor). This means we can work toward a system where the application (unikernel) has least privilege by default.
With Solo5 [1] we're working towards defining such an interface. We have a way to go yet [2], and are still very much in the "early adopter who knows what a unikernel is" stage, but the possibilities are exciting, not only from the application security point of view.
[1] https://github.com/solo5/solo5 [2] I'd be the first to say that the current Solo5 interfaces are suboptimal, but we're working on it: https://github.com/Solo5/solo5/pull/201
I know unikernels are single process, but are they also intended to be single core? If not, without the kernel scheduler, what is providing that function to the unikernel? Is it just an optional module, say in the case of rumprun?
The API specifics are probably different but it's just setting some registers and the the program counter on that CPU core. The code likely looks very similar to a context switch.
Besides languages, though, plenty of complex applications have their own schedulers. Databases, for example. Postgres-as-a-unikernel would make a lot of sense, because it's already doing its own equivalent to everything that OSes do.
If your program is complex enough, it likely will already have an "optimized" form of internally-managed shared memory/message-passing multithreading (with core affinities et al) to allow it to allocate tasks to cores better than the OS can without the "insider knowledge" of what it's trying to accomplish. At that point, your program will end up with a scheduler "inside" whether you set out to build one or not.
On the other hand, if your program isn't that complex, then it's very likely you'll get better performance out of your code by just making each process into a single-core unikernel and then packing instances onto however many vCPUs the hypervisor host offers.
In the case of the JVM, green threads were dropped a long time ago, and under the current paradigm, native threads are used (on all platforms?). So doesn't this need to be written into the JVM? Not impossible of course (or provided by the unikernel wrapper as it is in OSv).
Similarly with Databases, Oracle and Postgres both use a process per connection client/server model. So a unikernel for these would be impractical without rearchitecting those to run in that kind of environment.
I was initially very excited for Unikernels, but because of this question (and others), I've really started becoming less enamored by them. I'm sure there are many places where they can be put to use, but for most of the software that I'm familiar with, there are still a bunch of kernel services that are quite handy.
At what point of building these services into your unikernel/userspace application do you cross-over to basically building a new kernel. It really feels like the micro-kernel architecture would be a better fit in many of these cases. And yes, I understand that we can think of unikernels as VMs based micro-kernel services, but they have more isolation than traditional variations specifically allowing them to be used in AWS, etc. But still...
IncludeOS is rather liberal and has rudimentary multicore support. There is no preemption however so the "threads" will have to yield if there are more threads than cores. So more like fibers than threads, really.
The hypervisor typically doesn't care. So if you are using multiple cores in your application it will schedule and de-schedule your VM as per the regular rules.
IncludeOS is capable of booting on bare metal and we have some basic SMP tests passing. This is currently mostly of academic interest but if you don't mind having no pre-emption you can, once we improve our hardware support, have an application that runs on multiple physical cores. Perhaps this might be of interest to people with very strict latency requirements.
(yes, I know this already exists and it's container-based, but similar enough)
Why? Aren't they either usually on, or at most sleeping?
EDIT: no → on
The truth is, many developers cannot be trusted to write secure code much less provably secure. Kernels enforce some measure of security by design.
I'd expect unikernel devices to end up more secure: no shipping old Linux kernels full of vulnerabilities, and you're naturally guided towards using a safer programming language (OCaml).
However, there are other Unikernels. IncludeOS uses modern C++ so the Hello World looks like this:
#include <os> int main() { printf("Hello World!"); }
let () = print_endline "Hello, world!"
(from http://roscidus.com/blog/blog/2014/07/28/my-first-unikernel/)
Since a unikernel provides a self-contained system it could provide very clear and reasonably secure interfaces to the enclave.
On top of that users are currenlty moved away one additional abstraction layer from the kernel, to work with apps inside of containers (see Android, Kubernetes). The OS underneath becomes a static one-version-one-set-of-packages layer. That reduces the amount of people who can even decide to exchange a kernel for another, and these are mostly people who are rewarded for stability instead of experiments. Who would still like to write Android apps, if Android would start to crash more often or change its API drastically?
This is a battle that can't be won. Let's move on to one that is still undecided, e.g. which container engine will finally win. Although Docker seems to be the most likely favorite already at this point, I'd still like to see if nspawn will do a systemd move and just replace the alternatives.
Depending on the domain (embedded, IoT, industrial, medical, etc), tons of other kernels are used (and prevail). We're not necessarily talking web services here.
>This is a battle that can't be won. Let's move on to one that is still undecided, e.g. which container engine will finally win.
That's a totally different concern.
And no the container war is not a totally different concern. That's the current main concern which influences most people and how software will be administrated (and therefore developed) in the next 20 years.
I'm not really sure. I think containers (if we're talking of the Docker type) is a hype.
But in any case, totally different concern doesn't preclude it being important. Just irrelevant to unikernels.