avsm

If anyone wants to see a talk about this with a few demos of various use-cases (such as the JavaScript transpilation backend, and log servers), I gave a talk at QCon NYC that just went live on InfoQ yesterday: http://www.infoq.com/presentations/irmin

It's worth bearing in mind that the Rump Xen backend isn't well optimised at the moment, so there's quite a bit of low hanging fruit that could probably boost performance quite a lot. I think things like multi-page netfront, more careful rx/tx batching, and probably scheduler measurements will have an impact. The focus right now is on getting the individual applications to run reliably and robustly as unikernels when recompiled under Rump, and upstreaming patches where applicable.

OpenBSD/sun4v already has some support for virtualisation via logical domains. See: http://www.openbsd.org/papers/eurobsdcon2011-kettenis.pdf and the ldomctl manpage here: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/...

The Irmin support means that the DNS/unikernel state is stored in a Git-compatible format. Tried pushing it to GitHub via cron to see what happens? :-)

> If that's the main aim, then I don't see why they don't build their capability/crypto model into one of the unikernel OSes, specifically one of those that has the strong type safety and process isolation they are also aiming for.

They seem to be doing just fine with their current efforts, from which really interesting things like the MinimalT protocol has emerged. If I have one frustration, it's that they haven't released a reference implementation of it. I had a f2f conversation with DJB at Real World Crypto a few months ago and the explanation was completely reasonable: they are still working on cleaning it up and making it suitable for external consumption. I still want it today though :-)

David Sheets did some work on rebuilding portions of the protocol in MirageOS: https://github.com/dsheets/ocaml-dnscurve . This level of interproject coupling is just fine by me -- the whole point of library OSs is that we can pick'n'mix these things much more easily over the years, and that's working out fine.

Unikernels are a little different from Ethos. What unikernels need is a library interface to any underlying operating system, as opposed to a conventional syscall style interface that dynamically calls into a different privilege level. This lets the developer supply an appropriate set of "OS libraries" that fill in the functionality needed by the application, and no more, and for this to be resolved at application build time instead of execution time.

This implies that unikernels still need to have an underlying OS to drive their logic. The specific choice is wonderfully flexible though: right now we have numerous Xen-based unikernels, and via the Rump Kernel project MirageOS now also boots on KVM and bare metal targets as well.

Ethos has an interesting capability and crypto model that unikernels could map their distributed communication primitives too as well. In this world, the name resolution libraries used by unikernels could use CurveCP (or whatever equivalent version Ethos provides), and provide a multi-tenant, fine-grained capability-based, runtime for dust clouds of unikernels.

You're getting a 404 from nqsb.io? This: https://nqsb.io/nqsbtls-usenix-security15.pdf works fine for me. What browser/cache setup do you have? Does a `curl` work?

I left another copy of the paper on my homepage at http://anil.recoil.org/papers/2015-usenixsec-nqsb.pdf, but Peter's version or the USENIX site should be fine as well.

Definitely working on making it available to non-OCaml applications as well. The most obvious way is by using 'tlstunnel' to act as a TLS terminator to TCP, but also via inverted Ctypes to ship a shared library and C header files. More on that when it's working...

Sadly, we're not so flush with cash that we can significantly up the prize, which was itself a donation from the user community. It was quite amusing when some kind users donated Bitcoins into the piñata though :-)

We really like the idea of continuing the self-service security bounties, irrespective of their size. One of the nice things about unikernels is that it makes it easy to link in logic like this -- in a conventional OS, it would mean faffing around with kernel modules in order to safely seal the Bitcoins away, whereas here it's just normal high-level language code.

Incidentally, we're working on exposing a C interface to the OCaml TLS stack so that it can be used as a normal shared library as well. The approach is to use the OCaml Ctypes library (which is normally used to bind to C libraries from OCaml), but deploy it in inverted mode. This means that we expose a C ABI from OCaml code instead.

See https://github.com/yallop/ocaml-ctypes-inverted-stubs-exampl... for an example that exposes a C parsing interface to the OCaml XMLM library. The TLS stack isn't much more complex, but is pending us looking into libtls that are easier to expose than OpenSSL's. The s2n release here is thus nice and timely...

The 8.2MB can be reduced quite significantly with some compiler improvements. Here's a breakdown:

- Native code compilation still includes symbols. Stripping that reduces the size at the obvious cost of debugging ease.

- When an OCaml module is touched for a single function, the entire module is included at present. OCaml 4.02.0 included module aliases to make it easier to break down module hierarchies into less monolithic chunks. They work by exposing module equivalence in the signature, and reduced the size of Unix binaries in Jane Street Core by 90% in some cases; https://blogs.janestreet.com/better-namespaces-through-modul...

- Bytecode (while less performant) can be compressed more easily than native code. We've had DNS and OpenFlow servers that are less than a megabyte in size quite easily when compiled with bytecode.

- Dead code elimination requires some whole program optimisation, and currently only works with bytecode via the OCamlClean tool: http://www.algo-prog.info/ocaml_for_pic/web/index.php?id=oca... . Porting this to native code is on the TODO list, but needs some fiddling with frame pointers in the generated binaries to do a good job.

So the overall answer is that reducing the deployed binary size is now a compiler problem, since the unikernel architecture lets us provide it with much more information (configuration files and OS libraries as well as the application logic).

The live site is now running the full type-safe TLS stack all the way down to the TCP/IP stack and Xen device drivers. If it feels a little sluggish from your browser, this is because TLS session resumption hasn't been merged in yet (but is available to test at https://github.com/mirleft/ocaml-tls/pull/283)

The wg-windows@lists.ocaml.org working group has just started this week to sort this out properly. Now that OPAM itself is fairly mature on Linux/BSD/MacOSX, we can tackle this properly.

Please do feel free to join the group and give your 2 cents about the direction that this should take: http://lists.ocaml.org/listinfo/wg-windows

The interesting thing about the unikernel approach that we're discovering is how useful the portability of the resulting applications is. Because everything is structured as a graph of libraries, only a little modification can pull in a huge amount of useful functionality.

For instance, Martin's work on Rump+Mirage now pulls the whole safe OCaml stack into a bare metal environment: the type-safe TCP/IP, HTTP, TLS, DNS libraries can all be deployed onto any target supported by NetBSD/Rump -- which includes some pretty small embedded ARM devices that we couldn't boot on before due to the Xen requirement!

On the other side of the spectrum, Thomas Leonard has ported a good chunk of the MirageOS stack to JavaScript, and built browser applications using the Irmin Git datastructure library: http://roscidus.com/blog/blog/2015/04/28/cuekeeper-gitting-t...

We're working hard on getting all these pieces upstreamed at the moment, so it's not quite ready for primetime yet (the bare metal stack still has no network driver ported), but it's rapidly all coming together. I'm excited!

If you'd like to see the approach we're taking at OCaml Labs in order to build multicore, read KC's blog post here:

http://kcsrk.info/ocaml/multicore/2015/05/20/effects-multico...

The core idea is incredibly exciting (to us, anyway). Rather than baking in a specific multicore scheduler, we're allowing pluggable schedulers written in OCaml. They use algebraic effects to allow an independent scheduler to compose concurrency among OCaml threads. This will ensure that the OCaml runtime remains lean, and even allow applications to define their own strategies for concurrent scheduling.

> I'm most interested in seeing how jitsu and unikernels can turn the tables

The useful aspect about the MirageOS unikernels is that they use the pure Xen PV interface, which has almost no dependency on qemu. No floppy or block/net emulation, timers through the direct Xen shared_info page, and generally as "native" to x86 as Xen permits.

I'm greatly enjoying myself by using the Chrome developer console to see how the Irmin store (which is based on a git-like model) is logging and syncing the local actions.

It's intriguing that JavaScript compilation into a browser is a really nice way to debug complex logic that also ends up in Xen unikernel deployments...

That's true with a TCP connection and its three-way handshake. With a shared memory vchan across two VMs, establishing a connection is less than a millisecond (if that).

> I guess one benefit of Mirage's use of the OCaml module system is that we can experiment with multiple deployment approaches and use whichever one works best in practice. So there's nothing stopping me from running a Mirage application on Linux with raw sockets rather than as a Xen unikernel.

That insight is spot on. The reason we're spending so much time on the clean-slate MirageOS stack is that it's a codebase that will last the test of time. It makes it possible to build code that's more future proof against changes in deployment environments. It's Xen/KVM/vSphere today, JavaScript and Docker tomorrow, and probably Unix sockets forever, all from the same codebase. Our use of OCaml + functional programming also had a good trajectory with practical formal verification, so the future of the stack is exciting.

There are three ongoing efforts:

1) The Irmin Git store, which is designed to run both on Xen unikernels and in JavaScript. Thomas Leonard is working on integrating the latter, with very promising results for a React-style interface in the browser. Thomas Gazagnaire is integrating the Xen backend at the moment. Expect an announcement of both of these just as soon as they're ready.

2) The Rump Kernel project has very promising results with MySQL running as a unikernel, so we're working on integrating a MirageOS frontend with a MySQL backend unikernel.

3) The Ctypes library (0.4) that's just been released supports Xen compilation for OCaml bindings, so writing bindings to Sqlite would let you link in directly with a unikernel; https://github.com/ocamllabs/ocaml-ctypes . There's an ORM we wrote ages ago with this in mind: https://github.com/mirage/orm

So the db efforts are full steam ahead at the moment, but no fully released one yet for production. Keep an eye on the blog at http://openmirage.org, or drop a note to <mirageos-devel@lists.xenproject.org> for your intended usecase.

One interesting advanced prototype that Dave Scott from Citrix built is the XenStore database as a unikernel (that powers inter-VM communication) on top of Irmin. Video: https://www.youtube.com/watch?v=DSzvFwIVm5s and a presentation at the Xen Developers Summit: http://decks.openmirage.org/xendevsummit14#/

The term "OS virtualisation" is just fine here -- it refers to the support required to virtualise an operating system, which conventionally means the device driver, scheduler and memory interfaces. Providing such a layer lets any reinterpretation of "OS" still run above it.

Container-based technologies are far more nebulous in their definition, since it covers individual operating system concepts that don't quite migrate across distributions. FreeBSD jails, Solaris zones, LXC, MacOS X app sandboxes are all container-based virtualisation, but differ greatly in their details and specific interfaces.

Thanks, this are all really helpful pointers! It's particularly useful to find Lattice, which seems to be a simpler entry point than a full CF setup.

Note that both of the setups are quite different in how they boot unikernels. The EoX Zerg demo boots one unikernel per connection by having an HTTP proxy in the frontend spin up the VM.

Magnus' version builds a static set of unikernels whose total resource usage is known ahead of time, and boots them based on DNS lookups as necessary. Because there is a race between a DNS response, the first TCP SYN from the client, and the VM booting up to receive it, there is a helper unikernel called "Synjitsu" which terminates TCP connections and replays them to the MirageOS unikernel as soon as it boots up. This is very easy due to the OCaml TCP/IP stack in Synjitsu sharing its state via serialising it into an s-expression. More gory details in http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf

Now, imagine a version of Synjitsu that did this handover with TLS connections rather than just TCP. The recipient unikernel wouldn't ever even have to know the private key, just the ephemeral session key...

> I think the obstacle is that most folk are not prepared to give up their languages

s/languages/frameworks, and I agree. One step at a time -- this work is a foundational block that we're building on, not the ultimate solution.

That sort of design is what motivated Magnus' post that has been linked here. We've been laying the uniSaaS groundwork with a new Xen toolstack daemon called Jitsu ("Just in Time Summoning of Unikernels").

Here's the camera ready paper that we'll be presenting at USENIX NSDI in May: http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf

There's a lot more besides just the spawning to build a robust SaaS of course, but you can see the direction we're heading in...

> When requested to install a second package with overlapping dependencies as a previous package, it would try reinstalling all dependencies

This was unfortunately a mistake made by upstream Debian. Rather than reimplement the constraint solver badly, we use the Aspcud external solver. An upgrade of Aspcud changed its command-line interface, and OPAM 1.1.0 wasn't upgraded to OPAM 1.1.1 at the same time (which detected the new version of Aspcud and worked with it).

A similar issue (to do with the libdose interface) has been in the Ubuntu package ages and not addressed by upstream despite a detailed bug report: https://bugs.launchpad.net/ubuntu/+source/opam/+bug/1401346

> Searching online found recommendations to get the latest version from Github

What else could we recommend?

It's really frustrating to know that a source-compiled OPAM works fine, but that the packaged binary versions are broken and our only recourse is to wait six months for the next OS release of Ubuntu. On the bright side, the OPAM 1.2.x series has been very stable and is now making its way upstream, so this is hopefully a temporary issue.

The XenServer toolstack supports exactly this mode of operation in a pool of physical hosts via a 'host evacuate' operation that live relocates VMs away and brings them back once the host upgrade is complete.

http://docs.vmd.citrix.com/XenServer/5.0.0/1.0/en_gb/install...

It's worth noting that most of these bugs are right in the innards of the x86 emulation. Xen/ARM is a breath of fresh air, since they took the decision to only support the new ARMv7 virtualization extensions. This eliminate the need for qemu running in dom0 per VM and the instruction emulation plumbing.

We've got a distribution of Xen 4.4/ARMv7/ubuntu for anyone curious to try it out on a cheapo Cubieboard2 or Cubietruck over at https://github.com/mirage/xen-arm-builder (with prebuilt SDcard images at http://blobs.openmirage.org)

I really like some aspects of CommonJS/OCaml -- while digesting the whole thing, I've put together an `opam-boot` tool that does the standalone bootstrap so that you can build OCaml/OPAM applications without either OCaml or OPAM installed.

https://github.com/avsm/opam-boot (still unannounced and being debugging, but essentially working)

I'm liking the Merlin integration in CommonJS as well -- perhaps specialising it for js_of_ocaml applications would be one way to deploy it.

I can't really usefully answer that question without knowing more about what your purpose is.

If your purpose is to learn how to program in a functional style, then either language is fine. Other than that, the main difference between the two is purity/effects, lazy/strict, and type-classes/modules in Haskell and OCaml. That all changes the specifics of the code being written quite dramatically.

> I think most of the advantages of FP languages (as they exist today)

You seem to be describing Haskell with that rather overgeneral (and incorrect) statement above. OCaml supports mutation and explicit sharing just fine, and is strict by default. As for the rest, as you get closer to needing manual resource control, it just gracefully devolves.