My First Unikernel
roscidus.com
roscidus.com
So the dom0 OS (Linux) still determines which devices are exposed to the Unikernel. The Unikernel implementation is simple relative to a full "bare metal" OS because it only needs to support the Xen interfaces to block devices, the network, etc., and does not have to deal with disk drives, ethernet controllers, etc.
If you are running your own hardware, you're probably better off using something like FreeBSD + Jails or Linux + LXC (Docker). The Unikernel approach is more appropriate for situations where you want to deploy applications on Xen-based cloud infrastructure (Amazon EC2, Rackspace Cloud Servers) and do not want to waste resources or increase security risk by running a full OS. The physical servers at Amazon and Rackspace are already running a full dom0 OS (probably Linux), so running another full OS on top of that just to run your app is inefficient.
Xen also supports memory sharing among VMs if running in a hardware container, and Hwanju Kim added support for "PVH" mode to MiniOS recently which unlocks this functionality in Mirage. Early days on this work, but fun ones, since we aren't bound by the compatibility constraints of Linux containers, but have access to the same hardware resources.
Occasionally a good tradeoff, but a significant one.
[1] see table 2 at http://queue.acm.org/detail.cfm?id=2566628
Enough can go wrong in the packaging and distribution that it's a real risk.
At the end of the unikernel generation pipeline I get a little Xen VM spit out that I can run. How do I debug this guy? It's running on a hypervisor but is essentially in a different environment. I'd love to be able to take code written in one environment and be 100% sure it'd work in another, but that ain't going to happen soon.
Not true -- you can run Mirage Xen kernels directly as EC2 guest VMs by using their AKI feature to boot a custom kernel. Instead of a custom Linux kernel, just boot a unikernel and don't bother with user space :-)
Go to http://github.com/mirage/mirage and check out scripts/ec2.sh for a shell script that automates this (needs to be more user friendly, patches welcome)
If Ocaml can only handle one core and does not do SMP, how does it do in the cloud? Does this mean Mirage unikernels handle only one processor/core in Amazon and elsewhere?
+1 to this question? Any one experienced enough in OCaml to answer this?
A lot of popular web frameworks follow this paradigm and I would guess that most application code running "in the cloud" is in fact single threaded.
We are building this library to simplify this style of distributed programming: http://openmirage.org/blog/introducing-irmin
Other answer: we will be talking about our multicore ocaml implementation at ICFP in September. I still don't want to see it in Mirage though :-)
Are you assuming that vCPUs are scheduled or pinned? It wasn't clear from skimming the paper. I would agree that two levels of scheduling are bad but if that's the problem pinning should fix it.
The situation may change if you eliminate the hypervisor. My intuition is that a single multi-threaded process with work-stealing will be faster than separate processes or VMs due to better load balance (see PX vs. 1X dynos) and faster inter-thread communication (if your app has any communication).
The question of IPC performance is an interesting one. We've been building up a database of open source results that show wildly diverging results on different architectures. Surf through http://fable.io for it, or read this work in progress:
http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1.... http://anil.recoil.org/talks/fosdem-io-2012.pdf (fosdem slides)
The TL;DR of these numbers is that it's very hard to make firm performance hypotheses about IPC across architectures, NUMA and hypervisors.
So I am understanding correctly message passing will most likely by the paradigm you encourage?
And, less on point:
> Other answer: we will be talking about our multicore ocaml implementation at ICFP in September. I still don't want to see it in Mirage though :-)
Wait what what what? So it is ready for prime-time? (Note, I did not say production.) I remember reading about it on Jane Street, but thought there will still proposals and lots of theoretical and planning wrinkles to iron out before an implementation became reality.
I do not want to derail this informative post by you anymore, but do you have a link to more on that topic?
And I remember seeing mention of Irmin and not getting how ti fit into your work, asvm. Now seeing it as an answer to my question makes sense.
And thanks for your work on Real World Ocaml. I have decided to get back into programming, and some of my co-workers were taking CS classes and systems programming (ironically Harvard Extension School, I can't afford it but good for them) used OCaml. This reminded me to come take a look and I find your book, and the OCaml resurgence fascinating.
Maybe one done, I will look at the Haxe compiler and go: "it is not so complicated, I get that now." Only in OCaml could I see people doing such crazy shit.
The following may be of interest [1]. There's been more progress since and we'll be talking about it at OCaml 2014 [2].
However, optimizing around context switches and task preemptions is something you would usually do if your application is actually bound by IO/context switching, is extremely latency sensitive or when you are trying to squeeze the last bits of performance out of a machine. Why did you choose to build such a microoptimized system in a garbage collected language? Doesn't this defeat the purpose of the whole exercise?
I feel the need for more powerful types and built-in/standardized exception handling too, but since performance seems to be one of your major goals, wouldn't something like C++ be a better fit here? You'd get proper error handling and a good type system (with some tradeoffs) without a significant performance penalty.
On a sidenote, I agree that code written in a functional language with a strong type system tends to be easier to get right than bare C, but this doesn't imply that all low-level code is bug ridden and unsafe. In fact the linux kernel is one of the most stable and reliable pieces of software I've had the pleasure to work with so far. Suggesting there is a problem with the linux kernel because it contains "a large amount of C code in security-critical places" seems a bit dishonest.
OCaml isn't all that much slower than C++. To use the Programming Language Shootout as a rough esimation[^1], it can even come close to matching C++ in certain programs, and is rarely more than three times as slow. And of course—your type system will catch more errors and your resulting code will be much shorter (and in my opinion, at least, easier to understand.)
Finally: the article didn't say "the Linux kernel"—it said "Ubuntu." The kernel itself might be secure and reliable, but a running Linux system is much, much more than just the kernel. And while C can be security-audited, many of the properties that are important to verify in a C program come entirely for free from something like OCaml—e.g., an arbitrary piece of C code might not segfault given certain input, but a given piece of OCaml code definitely won't. So maybe a running Linux system is "secure enough", but a unikernel like this will have a much smaller attack surface and stronger inherent security properties with basically no extra work.
Disclaimer: I'm not the original author, I'm just speaking generally.
[^1]: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...
I agree this is a big upside of the authors approach. Less dependencies lead to fewer problems caused by external/upstream changes.
> OCaml isn't all that much slower than C++. To use the Programming Language Shootout as a rough esimation[^1], it can even come close to matching C++ in certain programs, and is rarely more than three times as slow.
I wasn't only referring to raw execution performance but also to GC pauses and GC overhead which I think are the bigger issue. The benchmark you linked tests for compute load so this doesn't really show up. Anecdotal point; in most real world apps I have worked on the GC was a limiting factor.
> Finally: the article didn't say, "the Linux kernel"—it said, "Ubuntu." The kernel itself might be secure and reliable, but a running Linux system is much, much more than just the kernel.
That's the beauty of having a kernel though. If one of those userland processes is broken it won't affect the whole system.
> an arbitrary piece of C code might not segfault given certain input, but a given piece of OCaml code definitely won't.
This assumes that the OCaml compiler/interpreter and the hardware are free of bugs...
With a Unikernel there is nothing else to affect. You are running a virtual machine anyway so even if you kill the kernel you only took down yourself.
> This assumes that the OCaml compiler/interpreter and the hardware are free of bugs...
They are not free of bugs. For most purposes you can consider everything to have bugs in it. However there is a big difference between a compiler/interpreter bug (which usually just makes your system run differently) and a service bug (which can lead to all kinds of problems).
With Mirage, it's all amortised in one consistent, fast GC! To give you a sense of the malloc vs OCaml GC trade off, see http://anil.recoil.org/papers/2007-eurosys-melange.pdf
Malloc and free list management is remarkably complex compared to a fast, simple GC. It would be interesting to build an OCaml runtime in Rust to experiment with these tradeoffs in a more controlled fashion.
Except you wouldn't, which is a major reason to not use C++.
Edit: as enduser pointed out, in addition to saving resources, this means there is no context-switching from kernelspace to userspace in your guest VM!
So it's like Docker.
Docker/cgroups/namespaces allow you to isolate multiple applications running in userspace on the same kernel to a very high degree.
This gets rid of isolation and multiprocessing altogether and runs a single application without any kernel at all (i.e. with just a very limited amount of support code linked in).
Put simply; the technology behind docker improves isolation between processes and this does the exact opposite.
For some applications removing this overhead might be interesting - e.g. databases tend to fight with the peculiarities in the host OS (scheduler, caching, disk access patterns).
The major upside of this approach IMO is the added robustness you get, since deploying turns into a very deterministic procedure, there's no underlying OS updates and whatnot to break your app.
Remember back in the 80s, when the BIOS was actually an effective hardware abstraction layer--giving you a defined interrupt to ask the BIOS to, say, write to a disk--and the OS was just for module loading and scheduling and policy-based security? (Not that DOS did either of the latter.)
Well, Xen isn't the new OS; instead, the domU is the new BIOS, and hypercalls are the new BIOS interrupts.
I really hope to see Linux redone (or another *nix created) in this "unikernel" style, where everything hardware-like or HAL-like is taken out, and instead things like filesystem drivers are implemented directly in terms of hypercalls.
I also hope to see UEFI reimplemented as a resident domU, such that a plain old desktop or notebook computer could treat its user OS as a container-image to be slung around, rather than having it "own" the hardware. UEFI actually already supports this mode of operation--allowing you to boot "UEFI applications" that keep UEFI around to provide BIOS-like functionality--but I don't know of a single OS that makes use of that, rather than overwriting the processor interrupt vectors and claiming all of physical memory for itself.
Interestingly, it looks like the easiest way to support Azure...
This is really the opposite direction things are going, especially in x86_64. PV is horrendously inefficient in 64 bit mode, because of the removal of CPU ring 1 and 2.
http://wiki.xen.org/wiki/Virtualization_Spectrum#Problems_wi...
HVM allows PCI passthrough if your CPU and chipset support it, which means domU now has direct access to the hardware, with no dom0/qemu layer to get in the way and slow things down.
Sounds like a lot of fun.
Thank you.
Am I right?
that said, it's still really cool, but it's not something i'd use, and especially not in production
The developer using this doesn't need to know anything about Xen, they just see a single address space system that runs their code.
i'm not saying it isn't cool, but it's a very round about way of doing something, and as such it becomes more expensive in development time and skill required
Your argument is actually "It seems very new. Not enough people know/use it. Therefore, I won't use it." -- which is fine, but please don't mischaracterise it in terms of increased complexity or expense.
all in all it's a nice trick, but it's not very practical, because if it was practice we'd all be using dos for our vms, since you get the bare metal, and a little bit of a environment to bootstrap from.
Frankly I wouldn't want a developer who's too dumb to learn OCaml anywhere near my production code. Is this thing new? Yes. Will developers take time (=your money) to get up to speed on it? Yes. But do you need "better" developers, long-term? I don't think so.
> Not only that, but you're making the project more complicated. i.e if the project is to create x, then you have to also create y first and, it's harder to debug.
Maybe a valid concern, but I remember very similar arguments from C++ programmers in the early days of the JVM. Turns out the JVM is rock-solid and nowadays has better debugging tools than those for C++. There's no reason that couldn't be true for this approach. Or if it's easy to make a multi-target project that builds both a linux binary and a unikernel image, then debugging would be no harder than it is for existing OCaml code.
> and you've also got the added overhead of running xen.
If you're already running linux-in-xen then this is reducing overhead. Even if you're not, it could still improve overall performance by reducing context switching, in the same way as user-mode networking stacks.
> it's a nice trick, but it's not very practical, because if it was practice we'd all be using dos for our vms
This sounds rather like "this can't be a good idea because if it was we'd have it already". It's only in the last few years that xen and the "cloud" approach have become so popular, so a lot of new ideas and approaches are still being found.
for Erlang, there's ErlangOnXen: http://try.erlangonxen.org/zerg
There are others, if you care to search for them.