Comparing Triton containers to VMs and bare metal servers | Hacker News Reader

Comparing Triton containers to VMs and bare metal servers

joyent.com

43 pointsxj921 comments

https://www.youtube.com/watch?v=Ll50EFquwSo Building and scaling container driven cloud infrastructure - Bryan Cantrill (Video about Triton)

I've been watching Joyent with interest for a while now, but I don't think I or my employer will be using their public cloud any time soon. Frankly, the pricing standard set by DigitalOcean a couple of years ago, and since followed by Linode (my and my employer's current favorite) and Vultr, is pretty hard to beat. I don't know how I'd justify 2x the price (not even counting bandwidth) when the performance of Linode is good enough. So I'll just continue to watch with interest. Now that SmartDataCenter is open source, maybe someone else will set up an SDC cloud with pricing that's competitive with DigitalOcean and the like. That would suck for Joyent though, so I shouldn't really wish for it.

there are about 60 mentions of the word "container" on that page, including a full chart of questions "Compared to Docker in a VM", such as "Where are containers run"?

But the real question that should be at the top of the file, or at least that table of questions, is "what is a container"?

So: what is a container? (As opposed to a VM.)

The term "Virtual Machine" has a variety of meanings, depending on who you talk to - Kernels are using LLVMs, Java runs inside a VM, and hypervisors create sandboxed full-OS VMs. The full-OS VM requires/provides its own binaries, frameworks, and DLLs; not having to depend on the Host to provide these.

A Container is like a Hypervisor, but instead of hosting a full-OS VM, the Container holds a Docker, which is just the code and data. Containers are similar to Application Pools or SQL Instances, and there are "Container Databases" which are much more self-contained than the traditional database. Containers depend on the Container OS for binary files, frameworks, and DLLs.

I like to look at it as a spectrum: Bare metal - Virtual Machine - Container

At one end, you're both running the binaries and scheduling threads. In the middle you're running binaries but not scheduling threads. On the other end, you're depending on the Host to do both.

Container is a chroot jail on steroids. VM is a QEMU process (also in a container).

how can you run a totally different architecture from just a chroot jail? It doesn't make sense. How are you "on the metal" if you're running some totally different kernel from what the chroot-jail-host is running? You can't both be running the kernel on the metal...

It might help to see my presentation entitled "Docker and the future of containers in production"[1][2]; it covers the fundamental building blocks here, and should answer your questions. (If that presentation leaves you looking for more depth on the LX branded zone technology on SmartOS, see my Surge illumos day presentation from last fall.[3])

[1] https://www.joyent.com/developers/videos/docker-and-the-futu...

[2] http://www.slideshare.net/bcantrill/docker-and-the-future-of...

[3] http://www.slideshare.net/bcantrill/illumos-lx

It's not a different architecture. SmartOS is x86 based, as are the Linux binaries.

What lx-brand zones do is present an alternate system call table to the binaries executed inside the zone. Those "lx" calls are compatible with the Linux system call table and get mapped and/or translated to illumos kernel system calls. Binaries don't know the difference.

So Triton is basically (simplifying) Solaris / SmartOS zones with the LX stuff installed?

It's that, plus cloud orchestration (i.e., SmartDataCenter) which includes sdc-docker, providing the Docker API and image management.

QEMU runs in the container, in this case. So you've got a whole "yo dawg, I heard you like VM's in your containers" thing going on.

I guess it's briefly described as a multi-tenant container system that isolates tenants by grouping them into parent containers? Maybe?

I'm not clear on the question.

The containers don't run a different kernel. They're just regular processes, in an fancy evolution of a chroot jail.

If you want to run something like Windows or FreeBSD, then you break out QEMU.

Containers hosting hardware virtual machines running Windows, FreeBSD, or others

Can't we just describe VMs as VMs and be done with it? So it's a container that has a VM inside it?

Yes, the QEMU process that underpins the hardware virtual machine lives within a container on the host. It means that a vulnerability in QEMU that allows for guest escape will land you in an otherwise empty container, rather than out on the host where other tenants are accessible.

But Triton is first and foremost about _container-based_ virtualisation: which involves no HVM at all!

Ah, that's something I hadn't considered. Good point. I guess the overhead will be minimum in that scenario.

Is it possible to do the same thing with cgroups or jails?

No, it's not -- a Triton container is a SmartOS container running an LX-branded zone.[1][2]

[1] http://www.slideshare.net/bcantrill/illumos-lx

[2] https://www.joyent.com/blog/triton-docker-and-the-best-of-al...

Edit: I think I misunderstood the question -- apologies!

Surprising that Joyent used an extremely similar name as Amazon's extremely similar product, EC2 Container Service (EC2 expands to Elastic Compute Cloud, so the full name is Elastic Compute Cloud Container Service)

http://aws.amazon.com/ecs/

The name is very similar but the products are not so similar -- ECS simply schedules Docker containers across the virtual machines you already have running on AWS. Triton schedules and runs containers on bare metal. Virtualization of course has performance and I/O overhead.

Now if only they fully supported AMD CPUs...

SmartOS runs just fine on AMD CPUs. KVM support for AMD hasn't been merged to illumos-kvm yet (mostly just due to time), but you can do your own builds with it (https://github.com/jclulow/illumos-kvm/).

But Triton containers don't run in KVM anyway. They run natively on the OS with no VM. So you can download and run Triton on AMD today, just as you can with Intel CPUs.

Reply on news.ycombinator.com