Improving Docker with Unikernels: Introducing HyperKit, VPNKit and DataKit | Hacker News Reader

Improving Docker with Unikernels: Introducing HyperKit, VPNKit and DataKit

blog.docker.com

292 pointssamber37 comments

docker's go-9p now makes for the 3rd implementation of 9p in go:

docker/go-9p https://github.com/docker/go-p9p

rminnich/ninep: https://github.com/rminnich/ninep

rminnich/go9p: https://github.com/rminnich/go9p

There's also the Andrey Mirtchovski and Latchesar Ionkov implementation of go9p, but all I can find is a dead Google Code link from here: http://9p.cat-v.org/implementations

We did try the other implementations first, but had a lot of issues doing what we wanted with them.

With lots of OCaml love it seems, from a quick glance through the source repositories.

And Mirage. This release is really awesome:

- It puts Mirage in use in real production systems.

- It strengthens the ecosystem around Hypervisor.framework. After xhyve and Veertu, this is really nice. So much for the meme that newer OS X versions only integrate iOS features.

They only took, like, the network stack from Mirage. Their virtualization library is xhyve… well, turned into a library.

Open sourcing our libraries is obviously just the start. We're really keen to hear about any other uses that people might have and move HyperKit in the direction of easy integration with higher-level frameworks.

For instance, direct ports of unikernel projects to run against the HyperKit kexec APIs would be really fun. Any takers for MirageOS or HalVM? :-)

We've started keeping a list of "help wanted" issues for anyone interested in getting started with hacking in this area: https://github.com/docker/hyperkit/issues?q=is%3Aissue+is%3A...

What was the motivation to create a new repo with a different name rather than maintaining a fork of xhyve?

We're taking it in a fairly different direction from upstream xhyve and bhyve, with the integration with other kits like VPNKit, DataKit (and soon, FSKit). We wanted to be able to diverge the build system in particular, and this makes it difficult to maintain a direct fork.

However, we're committed to upstreaming patches to their relevant projects where relevant, and so the HyperKit Git repository is as close as we can make it. If it diverges a lot in the future with file renames, we'll have to figure out the Git-fu to make cherry-picks easier...

> ...steering committee chair of the Commercial Uses of Functional Programming conference

From Anil's profile on blog and considering he is founder of MirageOS written in OCaml, it would be surprising if it were anything else than OCaml.

I wonder if we'll see a move towards getting Docker working on FreeBSD using either Jails or bhyve finally, since it talks about using bhyve hypervisor... That would be really great.

There is a port of docker that works with jails+ZFS (and the linux.ko syscall translator to run linux images) https://wiki.freebsd.org/Docker

There's also jetpack for the app container spec (rkt) https://github.com/3ofcoins/jetpack

I'm also experimenting with building a simpler system.

It should be possible to build a simpler port for FreeBSD with the recent Docker 1.11 release. It moved the container execution to containerd (https://github.com/docker/containerd), so that's where a bunch of the Jails logic would go.

I'm quite keen to see Docker running on FreeBSD so I can use it for my CI pipelines. I'm less interested in Linux emulation to run Linux images -- I'd like Docker support on FreeBSD to run FreeBSD images first!

What advantages would Docker give you over jails themselves?

Can't speak for anyone else, but while I think jails are "better" for a variety of reasons it seems clear there is far more market momentum behind Docker. More particularly, its toolchain. Given the choice between building myself (or relying upon a small, but talented, pool of devs) and drafting off tens of millions in VC cash and a much larger community I think some would prefer the ability to have a *BSD CI chain that is just a config setting in a Docker setup.

Containerd is just one part of what Docker does. There is also image management (signing via Notary), distribution (pull/push to the Hub), orchestration (Swarm and friends), linked services (Compose), etc. You could reuse all the rest of it while running containers in FreeBSD jails as the method of containment.

Seems like only a year ago Docker changed how it used Virtualbox to boot VMs using machine (and caused me endless amounts of suffering trying to figure out how to fix it). Now it would seem they are getting rid of Virtualbox entirely with their own VM...which needs contributions.

My understanding is not great, but it looks like they're bypassing a host VM entirely and using a shim (HyperKit) to talk to whatever virtualization the underlying OS supplies.

getting rid of Virtualbox entirely with their own VM...

Not really. This builds heavily upon Hypervisor.framework, which is included in new OS X versions. Hypervisor.framework is pretty cool, because it allows for building VM applications that are sandboxed and don't require kernel modules. As a result, Hypervisor.framework applications can even be distributed via the App Store (see e.g. Veertu).

I wonder if Vagrant intends to provide a Hypervisor.framework-based Linux backend. That would be cool.

Edit: Experimental plugin: https://github.com/oldpatricka/vagrant-xhyve.

The drawback is that now VM manager is at mercy of what Hypervisor.framework provides. For example, Veertu does not support redirection of USB devices into VM because the framework does not support it. That makes it rather useless for my usage and I stick with VirtualBox for now.

Of cause, this is not an issue for Docker as the framework covers all its needs.

I'm running the new beta native Docker for Mac, it is really nice. You install from the DMG, enter your password, and you're done.

No more running 'eval $(docker-machine env dev)', no more having to delete the VM and rebuild it, etc.

Have your encountered any major bugs with Docker for OSX? I've been running docker-machine without any major issues for months but have noticed in the last few weeks that I need to reboot the VM lately due to weird caching issues.

I have constant connection problems between the Docker CLI and the Docker VM running Docker for OSX :/

In recent updates? We have changed how this works in the last few releases, we hope it will be more stable.

I had this also; will re-try a more recent version.

Please do let us know if it persists, to beta-feedback@docker.com or on the forums.

I had it crash a couple times when it printed lots of text from the container to the terminal. I haven't had that happen since the update from today, though.

Thanks for the feedback.

This: https://forums.docker.com/t/file-access-in-mounted-volumes-e...

Mounting sockets as volumes no longer works. This broke a few things I was doing with sibling builds (-v /var/run/docker.sock:/var/run/docker.sock)

Wow that sounds awesome! Are there any estimates of when this will be an open beta for everyone? Very excited to try it out!

Docker ID: shuber

I'm not sure, I was on the waiting list for a while, I've been in the beta for about two weeks now.

Docker is still pretty bleeding edge, and was never designed to work with Mac OS X or Windows from the get-go.

The fact that the proposed experience for OS X and Windows is so good is a testament to how hard the community and the Docker team have been working! It's natural that there would be some growing pains.

So you're saying it's really bad and that's why it's so good.

Very interesting work. I find go-9p quite fascinating and think it could really have broader applications. Docker if you see this, I actually think you're on to something for microservice development thats native to the docker world. I've been trying to come up with ways of replicating the unix philosophy around programs that do one thing well and the use of pipes but was always limited in my thinking in terms of http, json, etc, etc.

My advice, as a guy who's currently building something in the microservice space, explore this further. Spend some time building fit for purpose apps with this and see where it goes.

I guess I just want to know the take away. eg. Will consume less memory on mac.

If you're asking about Docker for Mac/Windows, an earlier blog post has a few details.

https://blog.docker.com/2016/03/docker-for-mac-windows-beta/

In essence, it's a much nicer experience using the app and since it's improving all the time (still in beta, fwiw).

Reply on news.ycombinator.com