Containers, Not Virtual Machines, Are the Future Cloud
linuxjournal.com
linuxjournal.com
See, the expensive part? it's not the ram, it's sysadmin time. That's why VPSs are killing shared hosting; it's actually easier to manage a bunch of VPSs than a bunch of people with shell accounts on the same box. Same goes for all the containerization technologies I've tried; it's easier to just throw xen/kvm on there and pay for the extra ram than to manage one user stepping on resources required by another user.
For me? that was the beautiful part of switching from jails to Xen; suddenly, I could just ignore customers who tried to burn all the cpu and ram they could. Now, if I can figure out how to do the same with disk? I'll be a very happy man.
That, at least, is where I'm putting my money; in more ram.
It doesn't work like the CPU scheduler; which is to say, I can't set ionice on all the guests and expect good performance on the light-use guests while a heavy-use guest is thrashing the disk. (I can expect the good performance for the light-use guests when the resource in contention is CPU time.) It's possible that either ionice isn't a good enough tool, or that I don't know enough about it... but I think it's far more likely that the variable nature of disk throughput makes balancing requests harder... (e.g. how many IOPS a second I get varies a lot from second to second, based on where the sectors I want to operate on are in relation to the read/write head. How much disk bandwidth I get varies even more.)
I am pretty curious about how much this situation will change with SSD storage.
[1]http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting...
And that's why I like containers, because the sysadmin time of updating and patching all VMs I own (and manage for friends/clients) is huge.
Yes, I should probably use chef/puppet. The potential incompatibility issues with automated VM upgrades are probably the same as upgrading the container's OS. But mentally it feels more risky.
There is a key difference between your circumstance the person you are replying to: you are managing a farm of your own VMs so that admin time is your's to find. If the VMs are sold to thers so you are just hosting them, then the VM updates are someone else's problem and the admin time that becomes your problem is dealing with people whose VMs that are (accidentally usually, but not always) abusing shared resources. This is something you don't have as much hassle with when all the VMs are your's: you have presumably planned the likely resource needs carefuly and if something out of the ordinary happens you won't answer back to yourself like someone who has bought a cheap VM would answer back to you when you tell them they seem to be doing something wrong.
Selling containers/VMs to others is a very different kettle of fish to running you own private farm.
How does salt compare with puppet?
If you ice ubuntu... JuJu is terrific for either VM or container use. I have used it to deploy all of open stack services into a single "super size" AWS EC2 VM where the open stack was configured to use LXC as its virtualization. Worked fine for what I wanted. To back up or clone that open stack cloud only required me to snapshot that single EC2 instance.
Remember that those machines all have slightly different software on them, so it's not simple to automate. Unless you're manage systems with something (eg puppet) it's not just a PITA or a question of laziness - it's a real task that takes up hours, not a quick 15-minute job you run over a coffee break, and that's assuming no breakage.
Mind you containers don't help with this either...
Hi, I'm the author of the article.
You should check out cgroups. You can isolate disk access between services and containers using block I/O shares with cgroups.
The way shares work in cgroups is that there's unlimited access until the resource comes under contention. Then, the kernel schedules activity based on which cgroups have shares remaining. This allows scheduling to be "fair" in the sense that no cgroup always wins; access just becomes proportional.
We use block I/O shares (and other cgroup methods) for isolating containerized MySQL instances on Pantheon, the platform I work on as CTO.
[1] https://www.kernel.org/doc/Documentation/cgroups/blkio-contr...
However, using xen or kvm solves that problem, by giving each guest their own ram that nobody else can fuck with. I wonder if then using cgroups for the remaining writes would work well? seems like the 'shares' approach might work better than the 'priority' approach that ionice uses.
It gets hard to say whether a shared page cache is a good thing or not, even though it may be unfair. I say this because I/O bandwidth getting exhausted is a huge issue.
For example, using cgroups to limit memory allocations for groups or processes seemed like a great way to fairly distribute memory. But, doing so forced such cgroups into swapping when they tried to exceed their limits, even when there was available memory on the host system. The swapping was so bad in terms of saturating disk I/O that we had two choices to maintain quality of service: (1) set the hard limit and OOM kill (or equivalent) within the cgroup when it gets exceeded or (2) not treat it as a hard limit and monitor usage separately. We chose the latter.
So, I honestly wonder, is it better to enforce separate page caches in the cause of fairness, even if it results in less efficient disk I/O? Or is it better to have a unified page cache and dedicate system-wide resources to increasing effective disk I/O bandwidth? (Do we focus on slicing the pie more fairly or increasing the size the pie while cutting sloppily?)
In my case, (well, really in the flat-rate multi-tenant case in general) fairness is what I care about, more than overall efficiency. From a business perspective, it's okay to not be all things to all people, if you are a cheap-ish flat-rate service, it's okay if your heavy users don't get as much performance as they'd like, as long as the reasonable usage customers still feel like they are getting what they are paying for.
My core customer, the hobbyist who wants something like a shell for IRC idling, personal mail server, DNS for personal domains and a place to experiment or run a development project... generally buys enough ram to cache all the disk they normally access. I mean, 512MiB goes a long way for your average unix sysadmin. So even if the disk is getting thrashed by heavy users, light users? once their pagecache is warm, they have a fairly responsive system for everyday sorts of things. Under Xen or something else that doesn't share pagecache? once the pagecache is warm, it stays warm. They can login a week later (assuming they aren't running a bunch of background stuff that's reading/writing disk and churning pagecache) and their /etc/shadow is still in pagecache, and the login is pretty fast.
Back when I was using FreeBSD jails? if the user hadn't logged in for a few hours, the /etc/shadow in their jail had been flushed from cache, and they had to read it from disk again. They were getting terrible service.
From a economic standpoint? in general, if you have a flat-rate service, your light users are where the money is. They pay just as much as your heavy users (as it is flat rate) and they use fewer resources. And really, I think it's fair that if you are a light user and you are paying just as much as the heavy user, you should promptly get your resources on the occasions that you do need them. Heavy users on flat-rate services are going to have less than perfect experiences, and this is fair too, I think, so long as expectations of service level were set ahead of time. Heavy users use flat-rate services because it's cheaper than pay per use services, usually... but the downside of that is that they then only get as much resources as they can get without impacting the light users.
Now, in a pay per use model? the opposite is true. In a pay per use model (like, say, google app engine) the heavy users are the real customers. The light users are sales prospects. So yeah; in that case, you focus on the heavy users (and charge accordingly.)
Personally, I believe this is why shared hosting is seen as so far inferior to platform as a service; shared hosting is usually billed flat-rate, so the service provider really only has incentive to look after the light users; they are better off if the heavy users go elsewhere. (and really, for five bucks a month, what do you want?) - whereas platform as a service usually means you get billed per use; in that case, it makes a lot of sense for the provider to focus almost all it's energy on making things better for the heavy users. Of course, they also charge those heavy users accordingly.
Like the author I think containers are a much better answer than VMs. That is particularly true in a "web 2.0" world. I wrote up a bit about that on http://ops.blekko.com. In an architecture where the "app" is effectively a schema across a bunch of platform services you can achieve much higher machine utilization with containers.
At some point I hope the team at Google writes up their results for publication.
So if you have a business where some of your customers want custom, bleeding-edge kernels, and others mainly value stable systems with high uptimes, and everyone wants to run a different distribution's kernel, then containers aren't for you.
I guess I'm trying to say that VPS will always have a niche in the market. The parent lives in that niche. Containers offer fundamental resource-efficiency advantages and may capture large markets from Xen and Xen-like technologies over the next few years, but I disagree with the article's implication that Xen will eventually become extinct. As the parent notes, RAM is cheap, and the difference in costs between container and Xen will give the former an advantage, but never be big enough to drive the latter completely out of business, since the difference is small (on an absolute scale) and Xen can do a few things that containers can't.
I came to the same conclusion nearly 5 years ago:
http://clubtroppo.com.au/2008/07/10/shared-hosting-is-doomed...
I overreached though. The thing that keeps shared hosting alive is the gigantic momentum of PHP apps like Wordpress, various bulletin boards etc.
>I overreached though. The thing that keeps shared hosting alive is the gigantic momentum of PHP apps like Wordpress, various bulletin boards etc.
I think the thing that keeps shared hosting alive is that most people don't want to be SysAdmins. VPSs don't remove the need for a sysadmin's services, they just push it down to the customer.
From that perspective, shared php hosting is not only more expensive to do well (vs. selling VPSs) it provides more value to the customer (when done well) - unfortunately, market expectations right now make people expect to pay more for a VPS; the net effect being that most shared hosting services have underfunded admin departments and are thus, kinda crap.
I mean, even as someone who sells VPSs? I want the shared hosting market to be stronger. Selling a VPS to someone who is not able or willing to sysadmin a linux box is not fun or profitable. It's a bad day for the customer, and it's a bad day for me. And it's reasonable to want to not sysadmin your own server; it's a lot of work. We need some place for those people to go.
Further, most of the tools I've seen? webmin and the like? The thing is, it works great... Usually, for a while. Then sometimes it breaks, and it's a huge emergency, as the customer had it working just fine for a while; the customer was tricked into thinking they knew what they were doing, and now they've got an emergency on their hands, and they don't even have the training that installing the apps by hand forces on you.
So yeah; the thing I see displacing shared hosting? not VPSs, but another kind of fully managed service. See, as you point out, the difficulty of managing a shared hosting host goes up with the number of customers on that host in a kindof exponential way. What if you made something like shared hosting (where the user doesn't have root access, and the host is managed by full-time sysadmins) but ran one user per host, and run each host in a VM or container or something that is managed in a largely automated way?
Spend the ram/cpu overhead to better compartmentalize users and take the exponential increase in difficulty out of adding more customers to a particular bit of hardware? It's something I've thought about for a while. Problem is, the temptation to give users root on their VM would be very strong, and once that happens? well, managing thousands of hosts isn't that big of a deal... managing thousands of sysadmins? yeah. Not a job I want. Either I manage the host or you manage the host. Sharing root is a difficult, high-touch thing that requires a lot of communication.
Edit: I believe that many of the platform as a service providers operate like this (e.g. your stuff runs in one or several VMs that are not shared with other users, but that you don't have root on.)
Considering how cheap platform as a service is for the light users, I wonder how that model hasn't destroyed the flat-rate 'shared hosting' model? Perhaps one of the platform as a service providers needs to offer a flat monthly plan (perhaps that has the option of degrading /or/ interrupting service when usage exceeds the agreed-upon threshold, sort of in emulation of the "unlimited" providers.)
It's that if you have an app that targets bare virtual metal, then you can push a lot of admin problems upstream to the app developers.
In particular: updating. If a single app controls the whole VM, you can drastically improve the likelihood that a single update mechanism will work properly. You can even automate it.
Similarly: configuring components. Right now the lowest common denominator software environment that is targeted is shared hosting. In shared hosting I cannot adjust the database, the web server and so on. So no account is made of how to make self-adjustability a part of the architecture.
And so on.
My point is that we should stop thinking about apps that we install on linux that are on VMs vs apps that are installed on linux in shared hosting environments. It's that the app should be conceptually be all that runs on the VM. That nobody installs "the stack", they install "the app".
I think it would still be a distinct niche apart from managed systems.
Edit: a re-reading of your comment suggests that, I think, we are groping towards a common concept, but from slightly different directions. If you're interested, I'll email a project proposal I wrote in 2010 about this very topic. See my profile.
This is interesting. I mean, if you can get it to work. I'm bringing up the point and click admin tools, though, because that's the best comparison I can come up with that actually exists that I've seen.
The thing of it is, I think that most things do eventually break. (I mean, like the kids in this thread talking about how easy an 'apt-get update; apt-get upgrade' is - it works great most of the time, then it breaks something horribly.)
There are things you can do to reduce the chance of that breakage, but eh, at some point you have to login and fix the goddamn thing.
Not that this breaks your model; I mean, if you are distributing, say, a blogging application; my dad is paying you $5/month and giving you login credentials on a standard VM of some type, he installs using the kernel/script you gave him, and the system uses your repos to upgrade nightly. If something does break horribly, you can login and fix it.
I still think that model is easier if you include the VM in that $5/month, and you run that VM on something you control.
I'd also be interested in the upgrade system itself. Systems which have atomic snapshotting would be more un-borkable, for example.
The reason I have stuck with them and not shifted to a VPS is because I would need a sysadmin - and quite possibly an on-call 24/7 sysadmin at that.
Although Webfaction don't promise to fix problems at the app level - they do help and in many cases have got me out of a pickle that I would have needed a sysadmin to fix if I was on a VPS.
Most of the edge cases you do encounter are due to poorly-written software. The most common thing is server software that depends on Linux's default overcommit behavior and malloc() never failing and then uses that to try to badly implement a memory-sensitive cache in competition with the kernel. Containers change overcommit behavior and malloc() CAN fail, so such software will eat all the memory in a container and barf. But there are hacks to get around this, and it's mostly an issue with POS rube goldberg machine web stacks like Apache/mod_php.
I've been following LXC for a while and see that some people are now using it in production. Once Linux has a main-line kernel container solution I think you'll see it used quite a bit more. Eventually the developers of crappy server code will be forced to fix their code to behave properly.
It's a common technique used by a lot of virtual machines and memory management code to get a large contiguous region of address space to work with (most JVMs work this way, SBCL does too). I'm not exactly sure why Xen can handle overcommit fine and Virtuozzo can't, the latter does need a custom kernel so in principle they have access to all the low-level virtual memory stuff.
http://smartos.blueprint.org/home/why-smartos-in-my-lab
http://dtrace.org/blogs/brendan/2013/01/11/virtualization-pe...
[1] Illumos/SmartOS proponents seem to go out of their way to avoid saying the S-word.
This abstraction was kind of leaky -- different processes typically share various namespaces, like filesystems, process tables and user accounts. As a result, virtualizing an entire OS became a way to make a tight abstraction of having an independent machine at your disposal.
The reason virtualization happened was because emulating the system hardware was easier than changing every OS to adequately separate their namespaces. Now that virtualization has the attention of OS developers, you have development like LXC / cgroup stuff in Linux, bringing isolation into the OS, where it should have existed all along.
AFAIK the mainframe world -- which, by the way, still exists -- has never gotten out of the early-computing mindset of having an expensive machine which must be shared between applications, so isolation and containerization has always been a feature of mainframe OS's.
But to say that it is "the future" and virtual machines are on their way out is a naive view.
There's plenty of reasons why someone might want a virtualized OS; customization.
What if my application is built to run on Debian but yours is built to run on CentOS? With container based virtualization we have to be put on different nodes. With VMs, we can share a node.
This is why VMs exist. This is why they will continue to exist.
Disclaimer; I work in the hosting industry.
And for those providers secure containers will have a much lower overhead than full blown virtual machines.
You know what the most popular hosting product is right now? It's not VPSes. It's shared hosting. Good old simple shared hosting outsells VPSes by a huge margin.
There's plenty of room for a variety of products, all the way from shared, to VPS, to container based to dedicated servers.
What I took issue with was the original article saying that containers would make VMs a thing of the past, which is silly.
We will still need virtual machines for different operating systems, different versions of the same OS, multiple instances to span multiple VM hosts for scaling/redundancy, and so on.
Containers will cut down on the number of VMs we have to deploy within an organisation, nominally... it will be a more effective way to manage resources. VMs won't go away, or even be diminished.
That's only true when the OS's in question require different kernels. For OpenVz/Virtuozzo for example their RHEL based kernel has long been the preferred one for a lot of people for non-Redhat Linux versions too.
I agree with you there are still lots of reasons people will want VM's though.
We've picked containers at work, but only because while we host customer sites, we run and manage all of them, and so have a lot of flexibility a typical hosting provider doesn't have.
Virtual Machines let us abstract away our hardware resources. They also made deploying new instances much easier than it was previously, so we started using more instances than we would ahve without it, just to provide configuration and resource separation.
The first part is still necessary - and we still need multiple instances to take advantage of multiple VM hosts when it comes to scaling and redundancy. The second part is what containers address.
I have a feeling we will end up mixing and matching the two as appropriate.
Containers are - depending somewhat on the container technology used - from a user perspective just like a VM that happens to be restricted to running the same kernel as the host and use less resources. Some container systems may leak more in terms of abstractions and/or might not be designed to be as secure, and so there's certainly scenarios where certain types of containers are not appropriate, but that's an issue of specific implementations.
In the systems I deploy, containers takes care of all of what you describe all by themselves.
True, using today's off-the-self software it is easier to customize your environment using full virtualization compared to containers. But it isn't an inherent limitation of the container approach. Containers do however, provide inherent cost savings compared to full virtualization.
I expect that in a short amount of time, usability for containers will continue to improve. See for example docker: http://www.docker.io/
> What if my application is built to run on Debian but yours is built to run on CentOS? With container based virtualization we have to be put on different nodes. With VMs, we can share a node.
With the cost savings afforded by containers, I expect developers will generally choose to write applications that are portable enough to work inside containers.
> This is why VMs exist. This is why they will continue to exist.
Virtualization is popular because it is a feature-rich, economical way to host. Containers are more economical and I expect will win in the end.
Then there's emerging technologies like rump kernels, which might permit endless kernel customization with very tiny overhead... even emulating other OSes.
Also, avoiding having to deal with Linux.
The costs in complexity, incompatibility and pure byte bloat of running Debian & all are going to eventually be their death knell.
See for example https://github.com/steeve/docker-opencv/blob/master/Dockerfi... for a completely hermetic build process which outputs a self-contained tarball which can be run as an lxc container.
If I could do it all with Solaris zones, I would. The main problem for a hosting company is they would be hard to sell.
[1] Mirage - http://openmirage.org (paper at http://anil.recoil.org/papers/2013-asplos-mirage.pdf)
[2] Zerg - http://zerg.erlangonxen.org
Containers may be useful for single-tenant partitioning, but no one has managed to produce an OS kernel without an order of magnitude more local root security vulnerabilities than the HVM or PV virutualization systems.
Most of the challenge is automating that process adequately, which is really nothing more than traditional code profiling meets existing policy-based kernel-level security toolkits meets containers meets development processes (ie. build/test/release).
After ~3 years' with LXC, I believe that most of the attack surfaces are reasonably well understood and/or not required for most code to function. Sure, there will be kernel-level vulnerabilities, but they won't be exposed to such a broad range of userland applications. With security you also get so much more insight, power and predictability with regards to resource profiling than with paravirtualized solutions. Real kernel-level metrics versus load! Think of the impact for operations/capacity planning.
So... short soundbite to summarize? Docker is a toy compared to where we are going.
I agree with everything you said about lxc, but I'm not sure how you reached the conclusion that docker is a toy. Since Docker is a portability layer on top of lxc, it automatically benefits from all the awesome work happening in the kernel.
Or do you just use "toy" as a synonymous for "userland"?
I am talking conceptually about the whole direction the lightweight virt/app-level portability (relative infrastructure independence) thing is going.
My point is that the impact that continuing developments in this general area are going to have on security, development and operations processes is probably extreme... and I believe seriously under-considered by the community at present.
In terms of potential collaboration, what I would really like from docker is a single page, frequently revised summary page of features and efforts within the docker world. That would help me to get in touch with people to share info if we are facing similar challenges.
Right now it seems I would have to spend a lot of time monitoring things; time which doesn't exist. (He says, off to a sitar lesson...)
1) An ad-hoc compilation of cool stuff maintained by Ken Cochrane: https://github.com/dotcloud/docker/issues/614
2) All sorts of interesting software projects distributed as docker containers: http://index.docker.io
3) A highly active IRC channel, #docker@freenode where many cool applications of docker are being discussed
Your example two, the zerg system, seems very similar to LXC. Other than the exoticism of Erlang, their only big feature vs. normal paravirtualization seems to be startup time, which they see as a groundbreaking change. Well, I regularly start whole systems in LXC in fractions of a second. See https://github.com/globalcitizen/lxc-gentoo
In summary, these two projects are not really any different to the topic under discussion, ie. LXC (~cgroups/namespaces). Well, except that unlike cgroups/namespaces they're not necessarily able to run everywhere Linux runs!
That's very different from an exo-kernel running on bare hardware. With Mirage, what the apps run on seems to be "para-virtualized bare metal", so rather than using the POSIX syscall interface as its ABI, it's using the paravirtualized "hardware" as its ABI.
That makes Mirage very similar to Zerg - it's the same concept, only OCaml vs. Erlang.
As for LXC, I agree LXC offers a lot of this, especially since you don't need to boot full Linux environments - you can spawn individual applications and fine-tune what isolation you need. Or you can even build applications that explicitly use cgroups themselves to isolate sub-processes. So their "spawn one Xen Dom-U per request" demo could similarly have been a "spawn one new process isolated in its own cgroups".
But these systems are appealing to me in that they bring VM level isolation with startup performance and resource utilization more in line with container systems.
That's particularly appealing if we start seeing cloud systems that can handle it. Imagine being able to create new EC2 instances at sub-second times...
1) I'm not sure if this article has accounted for the one massive achilles heel of paravirtualization (containers):
Corrupting the core OS by way of upgrade or patches, can take all containers all out.
2) I have used VMware since the early betas in 98. The motivation was simple:
Replacing servers sucks.
The day you buy or lease a server, in essence, you purchase it's death and dealing with a hasty or painful migration.
3) From my 7th or so generation of server migrations in doing complex hosting since 95, one thing is clear:
The single most valuable thing is a hard separation between the bare metal and the OS.
So much so that you can move the VMs at will anywhere, because, you just bought the death of your server the day it's brand new. I ran a CentOS box with a BSD VM as a firewall, with Windows and Linux VM's behind it to serve one project because each did their unique and highly valuable part.
Ultimately, for me: Paravirtualization not only uses the host os, but shares it between multiple configurations. This is in my mind still introduces risks that should be minimized, if not avoided.
I'm sure PV is improving all the time. It's simply newer to the game than what VMware started. If VMWare isn't for you, suites like ProxMox are serious options to consider. Hosting isn't somewhere to have preferences and opinions. If I'm wrong, I'll change at any moment.
Issues with PV might be decreasing, but I've been ultimately burnt by PV every time I used it in production for a prolonged time.
An example of where sysadmin time cost way more than a little bit of more ram for Virtualization: Parallels Virtuozzo, corrupted itself so badly during an update to the core OS that all the containers on the server were rendered inoperable. Parallels support had one of their high level techs login from Russia to manually install corrupted DLL files so the server would at least boot, and we could get the heck off it. At the end of the repair we were told the VM was no longer upgradeable and all updates would have to be done manually.
The customer had taken on PV at the behest of the hosting company who insisted it was amazing despite my concerns. Said hosting company also now has switched over entirely to VMware.
The unforgivable kicker of PV for my customer: tons of customization and creating re-usable containers were all lost when leaving, because moving to something like VMware meant having to set them up from scratch again.
The main thing that is a non-neogotiable for my uses is one container should not be able to affect another.
In this way, Linux is less prone to "corruption" as you may see it, but given your implied familiarity with PV, I think you know what I'm referring to with there being plenty of gotchas floating about, especially in regards to sharing or using multiple kernels.
Anyone who's run multiple systems originating from different sources and being moved between hosts over time, know the reality of running different kernels.
Linux PV is not 100% issue and risk free as you implied, it certainly gives you a nice boost for some trade-offs. They are just trade offs I am not prepared or interested in handling for a small amount of ram.
Technology just doesn't work this way. Things don't simpler over time (and you don't shed support for different kernels, OSes, better isolation, etc.)
Not to mention the fact that virtualization has gotten dramatically better since EC2 was launched, since that seems to be the benchmark here (they are still running Xen 3.X first released in 2005). Modern virt hardware+hypervisors is blazing fast and (I believe) will eventually outpace bare metal for specific workloads on big boxes due to the improved isolation for many performance critical operations (I.e. separate TLBs for flushes, smarter cache invalidation). Even if they don't reach parity, the gap certainty wouldn't justify giving up all the benefits of full virtualization.
> Containers have been mature for longer than the article implies (freebsd jails, Solaris zones).
Mature containers have been around since the the days of mainframes. The recent (say, last decade) disparity has been lack of mature containers on the platform of choice, which is Linux for most cloud-based projects. I don't know many people who would switch their projects to run on FreeBSD or Solaris just to use containers instead of VMs.
Also, the container types you mention don't support independent, fine-grained choices of isolate-or-not for networking, the file system, user IDs, process IDs, and scheduling of system resources.
> Modern virt hardware+hypervisors is blazing fast and (I believe) will eventually outpace bare metal for specific workloads on big boxes due to the improved isolation for many performance critical operations (I.e. separate TLBs for flushes, smarter cache invalidation).
It's unclear why you think such optimization would appear in hypervisors but not kernels, especially given how much more insight a kernel has into the workloads it directly runs compared to a hypervisor running a kernel running workloads.
Even if hypervisors achieve CPU performance parity for running systems, you still haven't addressed the memory and storage overhead (in terms of resident footprint) of running full OS images instead of containers.
> Even if they don't reach parity, the gap certainty wouldn't justify giving up all the benefits of full virtualization.
You're portraying virtualization as having benefits versus containerization while ignoring that containerization also has many benefits over virtualization, especially w.r.t. cost efficiency.
> Mature containers have been around since the the days of mainframes.
Citation needed.
I'd go as far as to say that there is no such thing as a mature container technology. The fundamental problem of containers is that mainstreams kernels are not designed to be multi-tenant. They support multiple users quite well but when you try to have multiple root users, badness ensues.
Even the best container systems today for Linux still have fundamental gaps.
> Even if hypervisors achieve CPU performance parity for running systems, you still haven't addressed the memory and storage overhead (in terms of resident footprint) of running full OS images instead of containers.
Re: memory, same page merging can eliminate a lot of that overhead when running mostly homogenous workloads. Re: storage using CoW not only makes provisioning instant (not 5-10 minutes) but also addresses the storage concern.
If you mean the overhead of running two kernels, well, actually using namespace has a fair amount of practical overhead too.
Of course, benchmarks speak louder than words here. No one has published a container based result for SPECvirt because I'm quite sure it's not faster than virtualization.
First, I define containers by their capabilities, not a vendor or kernel calling them "containers." The Linux kernel, internally, has no concept of containers; it merely provides the resource-sharing and security-isolation building blocks to implement containers. It's userland utilities like LXC that assemble the necessary blocks together to provide what administrators recognize as containers.
So, let's talk non-virtualization-based resource- and security-isolation.
Workload Manager (WLM) [1] has been a part of IBM z/OS since before it was even called z/OS. WLM implemented something like cgroups scheduling, in that existing utilization samples feed into future scheduling to ensure responsiveness and fair access to resources under contention.
The Resource Access Control Facility (RACF) has mapped allowed program access to resources since 1985 [2], which functions similarly to modern mandatory access control.
> I'd go as far as to say that there is no such thing as a mature container technology. The fundamental problem of containers is that mainstreams kernels are not designed to be multi-tenant.
It's your turn to provide some citations. You could argue that the Linux kernel has failed to achieve good multi-tenancy support, but saying it hasn't been designed for multi-tenancy just makes you look ignorant of the last five years of development on namespaces, cgroups, and general multiprocess resource scheduling.
> They support multiple users quite well but when you try to have multiple root users, badness ensues.
Containers aren't about having multiple root users, even though that's technically possible if you opt to use Linux UID namespaces, which aren't mandatory.
> I'm quite sure it's not faster than virtualization.
Citation needed, especially since you're arguing that more layers of abstraction (kernel + hypervisor + kernel) is at or above the same speed of fewer (kernel + cgroups/namespaces).
[1] http://en.wikipedia.org/wiki/Workload_Manager [2] http://www-03.ibm.com/systems/z/os/zos/features/racf/racfhis...
I am intimately familiar with eWLM and I think it's quite unfair to call it containers.
> It's your turn to provide some citations. You could argue that the Linux kernel has failed to achieve good multi-tenancy support, but saying it hasn't been designed for multi-tenancy just makes you look ignorant of the last five years of development on namespaces, cgroups, and general multiprocess resource scheduling.
I'll give you a simple example. Do dd if=/dev/sda of=foo.img (or even /dev/null) in one container and measure I/O performance in the other.
The page cache is a global resource and so far there is not way to isolate it within containers. Buffered I/O is pretty fundamental to all workloads.
>> I'm quite sure it's not faster than virtualization. > > Citation needed, especially since you're arguing that more layers of abstraction (kernel + hypervisor + kernel) is at or above the same speed of fewer (kernel + cgroups/namespaces).
cgroups aren't free. See https://www.berrange.com/posts/2013/05/13/a-new-configurable...
I already cited a widely published benchmark (SpecVIRT). You article cites made up numbers (5-10 minutes to provision a guest).
$ qemu-img create -f qcow2 -b template.img new-guest.img $ qemu -hda new-guest.img -enable-kvm
And I have a nearly instant guest under KVM. So I don't know where your 5-10 minute number comes from but it's clearly not correct.
Note that I'm not normally one to bad talk anything. Containers are fine for that sort of thing. But you're claiming that containers obsolete virtualization and that's just plain silly.
I expect better from you :-)
I feel like you're strawmanning me, here. I specifically avoided calling WLM "containers." I said they're building blocks that, combined with MAC-style service isolation (which also existed on 1980s mainframes), allow containerized isolation of consolidated services on a single OS image. It also wouldn't be fair to directly call cgroups "containers," though they provide similar building blocks for what I'm sure we agree are "containers" once configured by a tool like LXC.
I compared WLM to cgroups because both track utilization over a window of time and manage scheduling in a way that avoids starvation while enforcing other priority or fairness goals. The documented resource limitation and contention-sharing configurations for WLM [1] are undeniably similar to cgroups.
> I'll give you a simple example. Do dd if=/dev/sda of=foo.img (or even /dev/null) in one container and measure I/O performance in the other.
Linux kernel cgroups have supported block I/O limits and weights for fair sharing under contention since 2008. I'll give you the benefit of the doubt here, though, and assume you're actually trying to provide an example of breaking the page cache because you mention that immediately afterward.
> The page cache is a global resource and so far there is not way to isolate it within containers. Buffered I/O is pretty fundamental to all workloads.
Shared page caching among containers is also one of the primary efficiency gains when multiple containers access the same on-disk assets. All along the spectrum, there are trade-offs between optimization via sharing (which risks decreased predictability) versus isolated resources (which necessitate efficiency losses through redundancy). I don't suggest containers are special in this regard.
Regardless, this same issue exists for virtualization, especially once you get down to areas like the caches on RAID controllers or other buffering done on the host machine. Isolating resources and security always has overhead.
> cgroups aren't free. See https://www.berrange.com/posts/2013/05/13/a-new-configurable....
Of course they're not free, but your suggestion that they're heavier weight than hypervisors running kernels is both counter-intuitive and unexplained/undocumented in your post.
> I already cited a widely published benchmark (SpecVIRT).
There wasn't anything you "cited" about the benchmark other than its existence because you didn't provide any numbers resulting from benchmark to support your arguments.
So, your post is at the stage of hypothesis: you have a prediction and a way to falsify it. Please don't pretend the results have already come out in your favor.
> You article cites made up numbers (5-10 minutes to provision a guest).
The numbers in the article are based on wall time for provisioning servers from public images on the Rackspace Cloud and EC2 using the API. Your qemu-img example is contrived because real-world systems don't keep local images sitting around on the host machines to provision new instances, disallowing the twin advantages of local, high-bandwidth I/O and copy-on-write.
In contrast, container creation that shares existing host machine libraries and binaries is very fast, including in real-world deployments (like PaaS providers).
> Note that I'm not normally one to bad talk anything. Containers are fine for that sort of thing. But you're claiming that containers obsolete virtualization and that's just plain silly.
But you are strawmanning me, again. I never said "containers obsolete virtualization." I said that containers will be "the future of the cloud," meaning that they should come to dominate given how well they're starting to fit most cloud users' needs. That doesn't imply obsolescence any more than saying "ATMs are the future of consumer banking" implies that tellers are obsolete and have no role in any sort of transaction.
> I expect better from you :-)
Well, I'd hope so, given our time in college together. I just realized this, by the way. :-)
[1] http://pic.dhe.ibm.com/infocenter/aix/v7r1/index.jsp?topic=%...
OpenVZ. VServer. Many people did that in the old days. Before Xen it was all about FreeBSD.
Awesome! Although I don't agree with the thesis, any systems article is a great article :)
> Also, the container types you mention don't support independent, fine-grained choices of isolate-or-not for networking, the file system, user IDs, process IDs, and scheduling of system resources.
Granted, containers have gotten better over time. My beef isn't with containers, just that they're often compared to virtualization as if they're a substitute or replacement. I really disagree with that.
Containers are still one kernel, with a huge attack surface. Forget about other OSes (including other versions of linux -- system calls do change over time, e.g. -- epoll, inotify, etc.). It's still a single buffer cache, so policing memory (beyond RSS) and disk I/O is tricky. It's one thing for a platform like Heroku, but containers don't make sense for a general purpose, multi-tenant infrastructure.
> It's unclear why you think such optimization would appear in hypervisors but not kernels, especially given how much more insight a kernel has into the workloads it directly runs compared to a hypervisor running a kernel running workloads.
Re: The long-term performance of virtualization.
It's a different abstraction which I believe lends itself better to using all the resources of large physical machines. We're talking future here (so to be sure, we'll just have to wait and see), but as a philosophy it makes sense to me. The same way we hit the limit on cranking the frequency of CPUs and eventually starting adding cores, a single operating system won't scale linearly forever. (Not that Linux isn't doing a laudable job). The optimizations I mentioned specifically apply to the use of virt hardware (i.e. the VT extensions effectively give you multiple TLBs because they are tagged) because there are different set of semantics associated with how those threads of execution run on the hardware. The restricted physical interface presented to VMs can free the hardware by shifting some low-level problems of scale and co-ordination into software (effectively achieving performance by forcing a more scalable software architecture).
> Even if hypervisors achieve CPU performance parity for running systems, you still haven't addressed the memory and storage overhead (in terms of resident footprint) of running full OS images instead of containers.
There are techniques to reduce these overheads, but you've got a good point -- there is a price to pay for full virtualization. But fundamentally, I think this is part of the trade-off of better isolation. A common buffer cache in Linux will improve performance of containers, but it becomes very difficult or impossible to enforce exact resource policies.
> You're portraying virtualization as having benefits versus containerization while ignoring that containerization also has many benefits over virtualization, especially w.r.t. cost efficiency.
I acknowledge that containers can be more efficient -- I just don't think it justifies giving up the benefits that full virtualization provides. I also think that virtualization is valuable because it enables some additional technologies, but that's a different story...
Ultimately, I think it boils down to the fact that I just don't think containers and VMs are comparable. I don't see how containers could displace virtualization for infrastructure use cases beyond a platform for specific applications. The EC2+Heroku model makes perfect sense -- some people will run containers inside VMs on a general purpose infrastructure, some won't.
http://wiki.smartos.org/display/DOC/Tuning+the+IO+Throttle
http://dtrace.org/blogs/brendan/2012/12/19/the-use-method-sm...
In a VM, you don't share a kernel with other users, so the local root exploit doesn't buy you anything.
I prefer containers and think VMs are a huge hack for most problems. But as it stands now, the number of local root exploits vs say remote exploits on Linux is probably greater by a factor of 100 or so, so you're taking much more risk.
Security is one of the problems with Linux containers that still needs to be resolved. (see http://mattoncloud.org/2012/07/16/are-lxc-containers-enough/, Google "lxc security")
Yeah, but if you can run services in containers that don't need root access, you know have to exploit the services, find a privilege escalation exploit, and then run your local root exploit. If you don't give out root access in your containers, you can get the advantages of containers while minimizing the downsides.
Isolation strategies involving syscall filtering and mandatory access control (MAC) tools like selinux dramatically reduce the attack surface. For example, I know several recent root exploits were not possible to run on Fedora with selinux enabled.
People also discover hypervisor exploits at a reliable clip, too. Virtualization isn't a panacea of security isolation.
AFAIK you have to add SELinux yourself, and all the PaaS providers are probably doing something custom (or not). The newer thing seems to be seccomp filters (http://lwn.net/Articles/494252/), which are motivated by ChromeOS. I would like to see a comparison; from what I can tell seccomp is a lot simpler conceptually, although there are fewer user space tools for it.
The article didn't actually say "LXC" but it seems to be what most PaaS providers are using. When I tried LXC, while lighter than VMs, it also seemed too heavy, because you end up with 5 or 6 processes (starting with an "init") for every process you want to run. Using just raw namespaces and cgroups seems to be feasible although again there are few tools to do that. Apparently systemd has support, although I don't use any distros with it.
Docker is also a new thing, but I think it is just on top of LXC, so it is only as secure as your distro's LXC is.
From what I understand is that ZFS was so slowly ported to Linux due to the license problems. But why were Zones not quickly adapted?
Basically I think it is a consequence of containers/sandboxing being a very "commercial" technology, even though they are open source. The main users are hosting providers, and there's a significant amount of money in that business.
In the 90's there was a hosting "land rush" with all of these companies like 1and1 and dreamhost selling shared hosting on Linux. They were the ones that developed Linux VServer and OpenVZ apparently, and I think the pace was too great to get it into the mainline. Interested in any first-hand knowledge people have.
And in the 2010's there is a PaaS "land rush", with all of these companies building on AWS and other IaaS, while needing containerization like LXC. The OP's article is calling for increased support in distros -- I think the same lack of time for cooperation is happening. Heroku, Cloud Foundry, dot Cloud, ActiveState, etc. are all using the same thing essentially, but there's a big land grab, so they are all maintaining proprietary and complex user space configuration.
The kernel features like the various namespaces are just about finished trickling in I think; that doesn't mean they're secure though.
Businesses like treating applications like appliances. They want to start up a system and not touch it again. Migrating an application to a new OS release costs them a lot of money. A zone forces an upgrade - everything on the machine needs to be rolled forward to the new release.
Since businesses typically have a patchwork of software in various levels of support, continuous upgrades can be an expensive (and possibly impossible) task. For example, the application vendor may require a license purchase in order to get support for the new OS release. The vendor may even have gone out of business, and the replacement is still 6-12 months away.
In a full virtualised platform, you're not forced to continually pay that maintenance cost. You are free to store it as technical debt and pay it in a more managed fashion.
In the virtualization world we argue often about how to do things “right” and what the “industry standards” are. I started a series a while back to help define and explain virtualization and various related topics. Specifically I even talk about this subject ( http://ewams.net/?date=2012/01/04&view=The_4_Types_of_Virtua... ).
The main problem with saying VM’s are “no longer the future” is they are mainstream and have been for a while, so they probably are not not cool enough for hackers but they are not going anywhere fast. Containers have a place and full blown virtual machines have a place. The trick is finding which one will benefit your organization and for which purpose. Most likely both can work depending on the requirements.
The other topic I notice that is usually overlooked when talking about virtualization is the actual OS. Here in this article it discusses mostly Red Hat based distro’s. Some organizations strictly use OSX, others Ubuntu, others RHEL, while others use some sort of mix. Each is different and even for closely related like Fedora and RHEL, what works on one may not work on another without hacking it or some other black magic. (Or it may work just fine, you know how it goes). The other point of the OS talk is where is Windows? We can pretend to be ostriches but Windows server is out there and in large numbers. Lots of people make lots of money writing applications that work strictly on Windows (same for *nix). Just do not forget about it.
If we move forward with containers, and since the topic has come up at least twice in the last 6 months, let’s start to see recommendations on how to run containers. How to pick workloads. How to optimize applications and kernels to run them. What kind of hardware to run them on. How to code for them. Etc.
Another benefit for PaaS/SaaS is that you don't necessarily sell RAM or DISK, you sell an application/service and thus can take FULL advantage of containers, like: - more efficient resources sharing(on host) - dynamic resources re-balancing - quicker start/stop/etc
Containers can be used to offer IaaS, but as was already mentioned in this thread, a few edge cases (custom kernel or kernel module, any filesystems support, any firewall support) prevent it due to support/operations overhead.
However I'm not sure building a multi-tenancy hosting environment based on containers or zones is the best idea.
Suppose the tun driver has a bug in it which can cause a kernel panic under some circumstances and that someone running OpenVPN in a container or zone hits this condition. You can now wave bye bye to all the zones running on that host.
https://github.com/joyent/illumos-extra/commit/9412039a18f2f...
I think it should be possible, that I could program if my container cpu load is over 70% for more than 40 seconds to launch a new copy of my container in the same or other machine..
This lack of a service to scale containers it's the first thing that needs to be done to compete with common VM clouds.
FreeBSD Jails lack the same fine-grained isolation choices versus the base system that the Linux kernel exposes through namespaces and cgroups.
That's not to say that Jails don't capture most of the value I argue containers have, but they're a different generation of design.
Also, I wrote the article for the Linux Journal, which obviously affects the solutions explored.
cgroups offers hard limits for some things, like memory, but it mostly opts for a model using "shares" that determine the fractional access to resources versus other cgroups holding shares against the same resource.
For example, assume there's CPU contention. cgroup A has 10 CPU shares and cgroup B has 90. Processes in cgroup A will get 90% of the CPU time, but it will not starve cgroup A because cgroup A will still get 10%.
This shares-based model also has a major effect when there isn't contention. Shares-based resources are burstable. Even cgroup A (with 10 CPU shares) can use 100% of CPU if nothing else needs it.
This "burstable" nature can be good or bad. It's good in the sense that most users will probably get more CPU than their shares guarantee most of the time. It's bad because users can start expecting more than their shares guarantee and get a nasty surprise when resources get under contention.
It's time to drop some analogies.
cgroups are very much like a highway with an HOV lane (or more): anyone can go very fast when there's no contention. But, during rush hour, lanes get distributed as "shares" of the road to the HOV and non-HOV groups. Neither the HOV nor the non-HOV drivers get starved for road access (though responsiveness may not be equivalent, by design).
Traditional "nice" is like emergency vehicle traffic. An ambulance every now and then works fine as "-20 nice" traffic. But, if you filled the road with ambulances, it would starve normal traffic of roadway access.
RCTL is sort of like a person riding reserved right-of-way public transit. From the time the person hops on the train at point A to when they get off at point B, it will be the same duration any time of day. They don't get to go faster during low-traffic times, but they also don't have to worry about a significantly worse experience during rush hour.
Capsicum seems focused on intra-application isolation; I'm not sure how to compare it to other OS-level containers.
Runs ubuntu or Red Hat and uses LXC for virtualization. Last I saw it was <$100k
LXC is good for stuff like OpenShift (PAAS), but for "VPS" offerings the VM will continue to reign supreme.
If I understand this correctly, you would want to be able to rent container slices with the specific stack you are going to use, already set up.
meanwhile, containers are more efficient.
For example, many projects only need a container or VM with 512MB of memory and one CPU core.
"Just wanna email, word process, and some web and some gaming" has relatively recently gone from a kilobuck P4 couple hundred watt space heater to a hundredbuck tablet that needs charging every couple days.
Unless you all can find a way to make silicon more expensive or dramatically increase processing demand, all these elaborate schemes to dole out processor are doomed to something like the per minute long distance business model where eventually more money was being spent on detailed billing than on actually providing the service... When the TCO of an elaborate virtualization system and its provisioning and billing and firewalling system exceeds the TCO of just spinning up a ubiquitous cheap low power system on a chip...