Unikernels: Rise of the Virtual Library Operating System | Hacker News Reader

Unikernels: Rise of the Virtual Library Operating System

queue.acm.org

85 pointsbrudgers28 comments

The projects themselves have progressed since this article and MirageOS is doing pretty well [1]. There was also a recent HN discussion on an experiment to create (and run) one VM per URL, which was built via MirageOS [2].

Although there are many applications of the unikernel approach, a longer term goal that I'm particularly keen to work towards is the idea of distributed personal clouds. Enabling people to run their own infrastructure, without becoming sysadmins, would be pretty cool [3]. Of course, there are other ways to achieve this but unikernels offer certain benefits.

[1] http://openmirage.org (I'm involved with the project).

[2] https://news.ycombinator.com/item?id=9263146

[3] http://nymote.org/blog/2015/brewing-miso-to-serve-nymote/

Regarding #3, I'm working on a project [0][1] that might be able to achieve that. It's not a direct goal of the project, but could be a subset of the primary goal. The primary goal is to be able to take a system description and be able to run it anywhere, be it public clouds or private hardware.

The base image is CoreOS (so, not a uni-kernel, but still pretty sweet). You can search Docker Hub for containers and create multi-datacenter, (and soon) multi-provider clouds. I do think uni-kernels may be the way to go, but AFAICT the proper tooling just doesn't exist yet.

It's still in pre-alpha, so there are many amenities that I plan to add that don't exist yet, but it will launch clouds (what I'm calling a set of related CoreOS clusters), create fleet units, and start your applications.

If anyone's interested in helping work on it, let me know or send a PR. All the code is free (MIT) and on github (link on top-right of site).

[0] http://www.cloud-launcher.io

[1] http://www.cloud-launcher.io?simulator | Just a simulation; won't create machines; any credentials will be valid

I'm ambivalent about unikernels.

On the one hand, I applaud the effort to simplify the software stack and promote the use of a safe language all the way down.

On the other hand, I wonder if running lots of small OS kernels on a hypervisor is the best way to do this. Might it be better to simply run each application as a process on a conventional OS kernel, perhaps with containers for isolation? This post from Brendan Gregg argues pretty convincingly that virtualization imposes a substantial performance penalty:

http://dtrace.org/blogs/brendan/2013/01/11/virtualization-pe...

I understand that a key premise of the unikernel approach is that the millions of lines of C code in a conventional OS kernel like Linux are a problem. But in any real Xen deployment I'm aware of, Linux is running in dom0 anyway. So why not take Xen out of the picture and run each application as a process in a container on Linux, using raw sockets so you can still implement the TCP/IP stack in OCaml? For deployments on Amazon, I guess the unikernel approach has the advantage that you're running on top of Xen anyway, so you're eliminating a layer. But for running on one's own servers or an ARM-based device like the Cubieboard2, it seems to me that Xen is just complicating things.

I guess one benefit of Mirage's use of the OCaml module system is that we can experiment with multiple deployment approaches and use whichever one works best in practice. So there's nothing stopping me from running a Mirage application on Linux with raw sockets rather than as a Xen unikernel.

> I guess one benefit of Mirage's use of the OCaml module system is that we can experiment with multiple deployment approaches and use whichever one works best in practice. So there's nothing stopping me from running a Mirage application on Linux with raw sockets rather than as a Xen unikernel.

That insight is spot on. The reason we're spending so much time on the clean-slate MirageOS stack is that it's a codebase that will last the test of time. It makes it possible to build code that's more future proof against changes in deployment environments. It's Xen/KVM/vSphere today, JavaScript and Docker tomorrow, and probably Unix sockets forever, all from the same codebase. Our use of OCaml + functional programming also had a good trajectory with practical formal verification, so the future of the stack is exciting.

If you run a unikernel under an OS you can do very aggressive sandboxing (eg seccomp type 2 under Linux), as it does not really use any of the OS facilities, eg just read and write a raw socket file descriptor. So you end up with a pretty decent userspace security model, and small statically linked elf files as your container format. I am currently working on this, should be usefully usable soon.

> "But in any real Xen deployment I'm aware of, Linux is running in dom0 anyway."

It's still separated from the the domUs via the hypervisor. So an exploit in one place isn't easy to escalate into a breach of the whole system. This is a security benefit.

From the rest of your comment, I get the impression that you care more for speed/convenience (even the post you link to is about speed). That's a perfectly reasonable position and the examples you provide are perfectly valid for different threat models (i.e. running your own trusted code on your own trusted server). Deployments to the cloud (or large scale teams) do need to consider different threat models.

As avsm points out in a sibling comment, you can target numerous back-ends with MirageOS -- meaning that you can compose a system that suits your particular needs.

In general, I get the impression that the whole virtualization trend is just a way to re-invent process isolation with newer technology.

The process isolation is very useful but they've got a point, why we should run million lines of code until we get to the point to run a single web server? Have we ever asked our self why the operating system which its only responsibility is running a web-server should take 300 seconds to boot?

Sounds like a badly optimized install then. Unless you are running a disk check, or have a hardware raid controller, on every boot, getting a kernel and the minimal environment to run a web server up should take under 1/10 of that.

So with a whole bunch of additional optimisations (not out-of-the-box) you can get an order of magnitude improvement. Unikernels are at least another two orders of magnitude faster than that (~300ms), without any optimisations.

Upcoming paper: http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf

Boot time is rarely an issue when working with long running daemons such as web servers.

We run the millions of lines of code because web servers typically serve up web applications which in turn depend on a whole host of infrastructure being available. The millions of lines of code support all the possible hardware that you could throw at it (and so you can save some if you compile your kernel just for that purpose), and if you know exactly which services your web-app will depend on then you can prune even more.

Specialization always pays off in terms of efficiency and it always costs in terms of ease of expansion and adaptation to future needs.

Boot time of a server is somehow important when you dynamically spin new instances. If spinning a new instance were a sub-second process, you could allow the number of instances you run (and pay for) to follow the demand much more closely.

But here the boot time is a proxy metric of complexity.

It's great to have software that can do a wide spectrum of things. But your server most often does a narrow spectrum of highly predefined tasks. You often go to great lengths to obviate and disallow any other activity on it, for security reasons.

So it would be great if you took the universal software, somehow cut it to your task set, and could throw away the unused parts (which are 90%+). Unfortunately, this is not really attainable with traditional OSes. Unikernels allow you to do just that—with some quite noticeable limitations, as of now, though.

But it's not just process isolation. You also isolate the entire OS, so you can tune your TCP stack to match this application without affecting the others. If you go deeper than that you can even give access to the hardware and thus reduce latencies where you care for it and still provide sensible isolation between the different applications.

In the shallowest mode it is just process separation at the deep end it gives you a lot more than that.

I don't know for sure with Mirage, but OSv does not provide process isolation since everything is running in kernel memory.

Presumably you'd run one application inside such an image and between two images you do have isolation.

Correct, and it's fair to say that the application pretty much is the image - if the application exits, the VM shuts down. Configuration can be kept out of the image with cloud-init, and I basically treat all images as immutable.

The majority of my experience has been using OSv, running Akka, and it's yielding excellent results. Image sizes are small (<200MB), easy enough to upload to AWS and create a new AMI in a few minutes. Boot time to a running application is a few seconds. The only possibly downside is that OSv (as yet) doesn't support paravirtualisation, only HVM.

It's not even new. It has a 40-year history and unikernel-like systems were a major research topic during in the late 80s and early 90s -- except everyone was dreaming that we'd have thousands of tiny machines to run it on, rather than big machines with virtualization.

"Not enough machines" and "network gets slow before we even get to enough machines" were the two main bottlenecks. Virtualization solves them to some degree.

What doesn't help nowadays is the marketing hype that tries to oversell everything and the insane complexity involved in sustaining the whole thing, since virtualization on x86_64 is still, to a high degree, an afterthough that was grumpily patched on x86 along with everything else. The PR bullshit makes serious people outside the high-availability & distributed field discard this as fancy fluff, and the baroque technology stack makes the development pace rather slow.

It is somewhat disappointing to see the term "OS virtualisation" used to describe what is really still just virtualised hardware; albeit with a less traditional OS on top of the HVM hypervisor. It makes more sense to reserve the term to distinguish container-based technologies, where what is virtualised _is_ the operating system -- not the hardware it runs on.

The term "OS virtualisation" is just fine here -- it refers to the support required to virtualise an operating system, which conventionally means the device driver, scheduler and memory interfaces. Providing such a layer lets any reinterpretation of "OS" still run above it.

Container-based technologies are far more nebulous in their definition, since it covers individual operating system concepts that don't quite migrate across distributions. FreeBSD jails, Solaris zones, LXC, MacOS X app sandboxes are all container-based virtualisation, but differ greatly in their details and specific interfaces.

This reminds me of Ling/Erlangonxen, unfortunately Ling is not a free-software, it's opensource but not a free-software.

And the fact that your operating system is the actual language you're coding with, reminds me of the fantastic project by Niklaus Wirth, The Project Oberon. The project Oberon is a language and operating system you can learn more about it at http://www.inf.ethz.ch/personal/wirth/ProjectOberon/, http://www.inf.ethz.ch/personal/wirth/ and a video by professor Niklaus Wirth himself: http://bit.ly/1ClvKSX

How is Ling not free software? From its LICENSE¹, it's using the Berkeley DB/Sleepycat license, which is considered Free by both the OSI and FSF.

¹ https://github.com/cloudozer/ling/blob/master/LICENSE

Is there a project for a DBMS on MirageOS? I would relay like to try out MirageOS on a web shop project of mine, but can't imagine to start writing a database, or even a file system to store images..

There are three ongoing efforts:

1) The Irmin Git store, which is designed to run both on Xen unikernels and in JavaScript. Thomas Leonard is working on integrating the latter, with very promising results for a React-style interface in the browser. Thomas Gazagnaire is integrating the Xen backend at the moment. Expect an announcement of both of these just as soon as they're ready.

2) The Rump Kernel project has very promising results with MySQL running as a unikernel, so we're working on integrating a MirageOS frontend with a MySQL backend unikernel.

3) The Ctypes library (0.4) that's just been released supports Xen compilation for OCaml bindings, so writing bindings to Sqlite would let you link in directly with a unikernel; https://github.com/ocamllabs/ocaml-ctypes . There's an ORM we wrote ages ago with this in mind: https://github.com/mirage/orm

So the db efforts are full steam ahead at the moment, but no fully released one yet for production. Keep an eye on the blog at http://openmirage.org, or drop a note to <mirageos-devel@lists.xenproject.org> for your intended usecase.

One interesting advanced prototype that Dave Scott from Citrix built is the XenStore database as a unikernel (that powers inter-VM communication) on top of Irmin. Video: https://www.youtube.com/watch?v=DSzvFwIVm5s and a presentation at the Xen Developers Summit: http://decks.openmirage.org/xendevsummit14#/

> 2) The Rump Kernel project has very promising results with MySQL running as a unikernel, so we're working on integrating a MirageOS frontend with a MySQL backend unikernel.

Interesting. The big issue I see from the proposed model (spin up your application on-demand) is the need to retain a long-lived connection pool to a "traditional" container/machine with a standard relational database (the cost of reestablishing a connection every time would obviate any benefit of the scheme).

That's true with a TCP connection and its three-way handshake. With a shared memory vchan across two VMs, establishing a connection is less than a millisecond (if that).

Sure, but if you're using both a read/write master and a hot standby for read-only queries on another machine, you're going to need TCP.

My first thought when reading about MirageOS was that they are effectively cramming DOS into a VM.

Reply on news.ycombinator.com