Xen and Docker: Made for Each Other | Hacker News Reader

Xen and Docker: Made for Each Other

blog.xen.org

77 pointsferrantim27 comments

I don't disagree with this post, but neither do I see the point of it. Anyone who is deploying containers on EC2 is already using the two technologies side-by-side, not that it means anything earth shattering. Maybe I'm in the minority here but I continue to not get the constant comparisons between containers and VMs. They're not the same thing, and they don't solve the same problems.

I spin up a VM from Amazon and it lets me share an underlying hardware platform in controlled ways so that I pay only for what I need and can't affect the other people sharing it. That's the problem VMs solve.

But now that the VM is running I still have to configure it for my applications. It's just a virtual server, with a potentially large gap between that state and the state I want it in. That's the problem containers solve.

The confusion I see comes from the fact that many people use/used VMs for things that containers actually work fine for. For example, developers set up VM with tools like Vagrant and VirtualBox to isolate the dev tools and services used by a project in a development environment. Docker does replace some previous use cases for virtual machines, particularly ones that did not need the security and resource control provided by VMs.

Yep, I agree. VMs encompassed the problems that containers address, whereas containers do not encompass the problems that VMs are designed to address at least not in any competitive way. Containers are essentially a finer-grained way to manage dependencies on a VM than the VM itself is. So yeah, I guess I do understand the reason why the comparison gets made. I just don't feel it's a very useful one.

I don't not agree with you. :) When people can stop comparing containers to virtual environments the better off both technologies would be! If you want to squeeze efficiency out of your system, use them both, together but independantly.

Docker[1] and VMware[2] seem to agree with you.

[1] https://blog.docker.com/2014/08/docker-vmware-1-1-3/

[2] http://cto.vmware.com/vmware-containers-containers-without-c...

Xen is a great technology, and the community has accomplished much in terms of features and performance improvements. But I doubt the long term future is Containers/Docker inside Xen guests, instead of just bare-metal.

Reasons to pick bare-metal:

1. Price/performance. For many CPU-bound workloads the Xen overheads are negligible, but for I/O-bound workloads the overheads add up. The network code path, especially at 10 GbE speeds, is sensitive to any wasted cycles. For large sites with thousands of instances, these overheads mean less efficiency, and needing to run more instances, costing more for the business.

Can the future of Xen eliminate (or near eliminate) I/O overheads, so that their resource cost is near bare-metal? Put differently: will there be a future where all I/O paths in AWS EC2 access PCI passthrough, or some similar technology to bypass synchronous overheads? That would be great, but is this realistic? Bare-metal provides this level of performance today.

2. Debugability. Many environments are sensitive to latency outliers, and measure the 95th and higher latency percentiles. Root causing these is difficult on hardware virtualized environments, where latencies end at the hypercalls, but easier for bare-metal environments. Even with root access on both the host and guests, tracing I/O between them can be painstaking work, as the host can't see and trace inside the guest. (Is the Xen going to solve this problem? Eg, could I have a SystemTap script trace both guest application-level I/O requests and associate them with host device level I/O? I suppose it can be done, but will it?) On bare-metal with containers, the host can see all, and trace I/O seamlessly from guest processes to bare-metal devices. Currently.

3. Density or efficiency. Resources can be shared at a finer level of granularity with cgroups, than is really practical with virtualized devices such as CPUs. Unless you are only running one big Xen guest on a host.

Xen (and KVM) are here to stay: there will always be the need for different kernel support for legacy applications. But for very large sites with thousands of instances, the future (although it may take a while) for sites with many instances and are performance sensitive, is likely bare-metal + Containers/Docker.

If you're a small site, or you care less about performance and debugging, then there may well be merit in the work this article covers.

> will there be a future where all I/O paths in AWS EC2 access PCI passthrough

That's a business, not technology question. Possible today if I/O is priced for exclusivity to one workload, same as a dedicated server. Most AWS customers want shared workload (low) pricing.

> Debugability

In general, hypervisors provide more debug options than a single OS, since the host OS can introspect into the guest OS, even to record/replay guest OS execution, e.g. http://www.slideshare.net/mobile/xen_com_mgr/xentt-determini...

Real-time hypervisor scheduling is an ongoing research area, e.g. here is an automotive use case, http://blog.xen.org/index.php/2013/11/27/rt-xen-real-time-vi... & http://www.cse.wustl.edu/~lu/papers/emsoft11.pdf

> for very large sites with thousands of instances

Yes, containers work well for Google who doesn't need to worry about isolation. Most businesses are not Google nor are they running a real-time workload. With containers, SE Linux provides a necessary layer of defense, http://opensource.com/business/14/7/docker-security-selinux & http://blog.docker.com/2014/07/new-dockercon-video-docker-se...

Have you considered using Ubuntu MAAS?

He worked for Joyent, so I'm sure he's considered automated bare-metal deployment tools quite a bit. ;-)

BTW, I tried MaaS recently and it seems to work fine but I could not successfully customize the installed OS which made it impossible to integrate into my environment.

I might not have been clear: I'd want bare-metal as the basis for running Containers or Docker. The Xen article described the value of using a Xen guest as the basis.

Ubuntu MAAS could be part of that, but I'd want to be managing containers, not metal.

I think of Xen as a dying technology.

Lately IBM passed around a paper aimed at Amazon that showed how badly paravirtual machines behave compared to bare metal.

What they didn't note is that newer Intel chips have serious support for hardware virtualization, and that with hardware virtualizaton you get near bare metal performance. And they didn't note that the new instance types on AWS support hardware virtualization.

Paravirtualization was an important technology for a while because it was the fastest sort of virtualization ten years ago, but now it is the slowest.

I'm not sure if you're already aware and have concerns with it, but Xen offers several levels, ranging from full HVM to PV:

http://www.brendangregg.com/blog/2014-05-07/what-color-is-yo...

Xen and KVM development is funded by Intel and others. After virtualization algorithms (paravirtualization) are proven in software, they move into hardware. This has been taking place for almost a decade and continues. There is no conflict between S/W & H/W virtualization.

Paravirtualization bad, hardware virtualization good, is not an accurate view. You actually want a hybrid model to get the best performance. That's why KVM developers have put so much effort into the paravirtualized virtio drivers, for example. Xen has put lots of work into taking advantage of new hardware virtualization technology where it makes sense and continuing paravirtualization where it makes sense. See the two links below for great details.

http://blog.xen.org/index.php/2012/10/23/the-paravirtualizat...

http://blog.xen.org/index.php/2012/10/31/the-paravirtualizat...

Very informative, especially the second page, thanks for linking that!

If you have any other suggested reading re Xen / Virtualization I'd be very interested in them.

Try http://www.brendangregg.com/blog/2014-05-07/what-color-is-yo... and http://www.brendangregg.com/blog/2014-05-09/xen-feature-dete... .

I haven't seen that paper but in my experience with thousands of Xen servers, the CPU overhead is less than 5% (typically about 3% or so), not sure what parameters are discussed when talking about how badly VM behave (obv. there's I/O contention for several VMs on a server etc)

Always a good surprise to see new companies embracing Docker. I'd love to chat - nick@docker.com

Ya, because this is an appropriate place to plug for some +PR... how about you just contact them on their mailing list or something...

You actually think they are going to read this and think, "Oh shit! Nick wants to chat!"

I met Nick by this way, and my project got published in Docker blog.

We run each services in an LXC container. Containers are run inside a KVM that is connected to a bridged interface (one KVM per VLAN).

This setup is very clean and easy to partition, secure and maintain. So yes, light vs real virtualization can work well together!

Site down? It's not loading for me on Desktop or Mobile. From the title it sounds like an interesting read, I'll have to check back later.

Edit: Mwuhaha. http://webcache.googleusercontent.com/search?q=cache:upPfoeO...

its just really slow:

Xen & Docker: Made for Each Other!

By Sarah Conway – September 8, 2014

By Olivier Lambert

Containers and hypervisors are often seen as competing technologies – enemies even. But in reality the two technologies are complementary and increasingly used together by developers and admins. This recent Linux.com article talked about this supposed battle, noting however that developers are using Docker in traditional VMs to bolster security. Containers allow users to develop and deploy a variety of applications with incredible efficiency, while virtualization eliminates any constraints and/or exposure to outside attacks.

Uniting these technologies helps developers and system administrators be even more efficient. Let’s take a closer look at how to achieve this with Docker and Xen Project virtualization, and why we expect more and more organizations to use them together in the near future. This will also be a key topic at the September 15 Xen Project User Summit at the Lighthouse Executive Conference Center in New York City. Register today to learn more about enabling Docker in Xen environments for a truly open infrastructure.

xen-docker

Caption: Xen In Action: lifting Docker, which is lifting containers. I heard you like boats, so I put boats on your boat :) .

Who’s Who: What is Xen Project Virtualization?

Xen Project Hypervisor is mature virtualization technology used by many of the world’s largest cloud providers like AWS, Verizon Terremark, Rackspace and many more. Founded in 2003, Xen Project virtualization is proven as a highly reliable, efficient and flexible hypervisor for a range of environments, running perfectly from x86 to ARM.

It’s now completely integrated in the Linux upstream and is hosted by the Linux Foundation. The same big cloud users mentioned above also contribute regularly to the project along with many of the world’s largest technology companies, including Citrix, Cavium, Intel, Oracle and more.

Feature updates and broader community collaboration are on the upswing too: more commits, more communication, better integration, new use cases and simpler and more powerful modes, such as PVHVM then PVH, as outlined in this recent blog.

The core Xen Project team takes security seriously. The technology has also been battle-tested by many in the defense industry including the NSA. Xen Project users have benefited from this for years, and developers building, shipping and running distributed applications will profit as well.

XenLogoBlackGreen

What is XenServer and Xen Orchestra?

XenServer is a packaged product consisting of the Xen Project Hypervisor and the Xen Project Management API (XAPI) toolstack within a performance tuned CentOS distribution. It’s free and can be installed in just a few minutes; click here to download it: http://xenserver.org/open-source-virtualization-download.htm....

Xen Orchestra (XO) is a simple but powerful web interface working out-of-the-box with XenServer, or any host with Xen and XAPI (the most advanced API for Xen). Take a look on the project website to learn more. Both of these tool are of course free software.

What is Docker?

In its own words, Docker defines itself as an open platform for developers and sysadmins to build, ship, and run distributed applications. Consisting of Docker Engine, a portable, lightweight runtime and packaging tool, and Docker Hub, a cloud service for sharing applications and automating workflows, Docker enables apps to be quickly assembled from components and eliminates the friction between development, QA, and production environments.

docker-logo-370x290

Main Advantages:

    fast (boot a container in milliseconds)
    simple to use, even in complex workflows
    light (same kernel)
    container density on one host

The other side of the coin:

    all containers rely on the same kernel (isolation, security)
    less maturity than traditional hypervisor (Docker is still young)
    containers are using the same OS on the host (less diversity than hypervisors)
    some friction between developers and admins about its usage: not Docker’s fault, more a classical friction when you bring new toys to your devs. :) We’ll see why and how to cope with just that below.

Best of Both Worlds

An ideal world would:

    Let admins do their admin stuff without constraints and/or exposure to dangerous things.
    Let developers do their developer stuff without constraints and/or exposure to dangerous things.

Fluid Workflow

In other words, they’d be able to create really cool workflows. For example:

    An admin should be able to easily create a Docker ready VM running in a hypervisor, with the exact amount of resources needed at a given point in time (he knows the total amount of resources, e.g a VM with 2 CPUs and 4GB of RAM.
    He should delegate (with the same simplicity) this Docker-ready VM to the dev team.
    Developers can use it and play with their new toy, without any chance of breaking stuff other than the VM itself. The VM is actually a sandbox, not a jail; developers can create their containers as they need in this scenario.

Now you can easily imagine other exciting things such as:

    An admin can delegate rollback snapshot control to a developer. If he breaks the VM, he can rollback to the “clean” snapshot — without bothering the admin staff. Live, die, repeat!
    Need to clone the same container for other tests? One click in a web interface.
    Need to extend the resources of this current VM? One click, in live.
    Ideally, let a developer create its container from the same web interface.

Xen Orchestra: A Bridge Between Docker and Xen Project Hypervisor

So how do we do all this without creating a brand new tool? As you may guess, the answer is Xen Orchestra, which today achieves much of this. Updates planned for later this year and 2015 will deliver even more efficiencies between the two technologies.

What XO Does Today

    Adjust Resources In Live: You can reduce/raise number of CPUs, RAM, etc., while the VM is running! Doing this, you can grow or reduce the footprint of your Docker VM, without interrupting the service. Check it out in this short video.
    Snapshots and Rollback: Snapshots and rollback in XO are totally operational since XO 3.3. Check out how this works in this feature presentation. Coupled with Docker, this is very helpful. When your fresh Dockerized VM is ready, take a snapshot. Then you can rollback when you want to retrieve this clean state. All with just a few clicks and in a few seconds.

Coming Soon

    Docker-Ready Templates in One Click: This feature will be released this year. In a few words, you can request our template directly from your XO interface, it will be downloaded and operational in your own infrastructure with a Docker listening and ready for action,Iin the resources you choose to allocate (CPU, RAM, Disk). No installation: It works out of the box. Read more in this article.
    ACL and Delegation: The perfect workflow rest upon integration of ACLs in Xen Orchestra is our current priority. In our case, it allows VM delegation for your team using Docker; the VM can be rollbacked or rebooted without asking you. More info. here.
    Docker Control from XO: Because we can get the IP of a VM thanks to its Xen tools, we should be able to send command to the Docker API directly through XO. In this way, you’ll just have to use one interface for Docker AND Xen (at least, for simple Docker operations). And take the best of XO for both: ACLs, visualization etc. This last feature is not in our current roadmap, but will probably pop up early in 2015!

We-need-to-go-deeper_inception

Caption: Coming soon — deeper integration between Docker and Xen.

Conclusion

Docker is a really promising and growing technology. With Docker and Xen on the same team, the two technologies work in tandem to create an extremely efficient, best-of-breed infrastructure. Finally uniting them in one interface is a big leap ahead!

Any questions or comments? Go ahead!

By Olivier Lambert, Creator of Xen Orchestra Project

It seems you could do most of the resource sharing, separation and dynamic allocation with cgroups in docker (should it be implemented)

XEN mainly provides a stronger separation (or, like, KVM does that, too)

It's commonly missunderstood but Xen can also do much more then KVM because it can run systems purely para-virtualised.

This means there is no actual machine abstraction despite it using hardware accelerated extensions like VT-x and EPT.

The difference in overhead between a fully para-virtualised system and a process that is running in a container is going to be pretty similar other than I/O performance (which is a complicated discussion).

If you find this interesting lookup Xen unikernels, a fairly well known one now is Mirage OS. Which lets you easily write Ocaml applications ontop of a Xen unikernel.

Would that be simple as running a CoreOS image inside Xen? Does it make sense?

Reply on news.ycombinator.com