It is the spot between chroot and VM. Looks like a VM from the inside, provides some degree of resource usage QoS and does not require you to run a full operating system like a VM.
Another concept that is now also often automatically connected to containers is the distribution mechanism that Docker brought. While provisioning is an orthogonal topic to runtime, it is nice that these two operational topics are solved at the same time in a convinient way.
rkt did some nice work to allow you to choose the runtime isolation level while sticking to the same provisioning mechanism:
https://coreos.com/rkt/docs/latest/devel/architecture.html#s...
Seccomp only landed for Docker in about 1.12
https://channel9.msdn.com/Blogs/Seth-Juarez/Windows-10-Virtu...
(or Windows 10 isolated user mode, which seems kind of similar)
https://channel9.msdn.com/Blogs/Seth-Juarez/Mitigating-Crede...
as the article shows, this is not the point for Zones and Jails.
I'm also becoming more sold on the Kubernetes model, which relies on containers. Build your small service, let the system scale it for you. I don't have as much hands-on here yet, but so far it seems pretty great.
Neither of those are the same problems that VMs or chroot are solving, as I see it, but a completely different problem that gets much less press.
Signed jars are a little painful to use (you can't easily bundle them), but that's a minor issue.
Page sharing can also depend on the storage driver; overlayfs support page cache sharing and brtfs does not.
But there are more benefits like Jessie touches upon in her blog post, wrt flexibility and patterns you can use with multiple containers sharing some namespaces, etc. And from the perspective of languages that do not compile to a native binary the containers offer a uniform way to package and deploy an application.
When I was at QuizUp and we decided to switch our deployment units to docker containers we had been deploying using custom-baked VM's (AMI's). When we first started doing that it was due to our immutable infrastructure philosophy, but soon it became a relied-upon and necessary abstraction to homogeneously deploy services whether they were written in python, java, scala, go, or c++.
Using docker containers allowed us to keep that level of abstraction while reducing overheads significantly, and due to the dockers being easy to start and run anywhere we became more infrastructure agnostic at the same time.
And that's great news. Immutable deployment artifacts let us reason about our systems much more coherently.
So now we have the issue that you have lots of applications running on the same server, and how do we make sure the right version of some shared lib is on there. And that we won't break another program by updating it.
Containers solve that. No more worrying if that java 8 upgrade will break some old application.
So now every application stack is a static application.
...this applies only software developed and run internally, which is a small fraction of all the software running in the world.
This has literally zero to do with containers and everything to do with an automated deployment pipeline.
As a quick FYI: Those are not unique to containers.
As one very minor point, it turns vulnerability tracking in to an accounting exercise, which sounds like a good idea until you take a look at the dexterity with which most engineering firms manage their AWS accounts. (Sure, just get better at it and it won't be a problem. That advice works with everything else, right?)
One's choice of deployment tools may slap a bandaid on some things, but that is not the same thing as solving a problem; that is automated bandaid application.
And odd pronouncements like any given container shouldn't be long lived are... odd. I guess if all you do is serve CRUD queries with them, that's probably OK.
As a final point, I feel like the container advocates are selling an engineer's view of how ops should work. As with most things, neutral to good ideas end up wrapped up with a lot of rookie mistakes, not to mention typical engineer arrogance[1]. Just the same thing you get anywhere amateurs lecture the pros, but the current hype train surrounding docker is enough to let it actually cause problems[2].
My takeaway is still the same as it was when the noise started. Docker has the potential to be a nice bundling of Linux capabilities as an evolution of a very old idea that solves some real problems in some situations, and I look forward to it growing up. In the mean time, I'm bored with this engineering fad; can we get on with the next one already?
[1] One very simple example, because I know someone will ask: Kubernetes logging is a stupid mess that doesn't play well with... well, anything. And to be fair, ops engineers are no better with the arrogance.
[2] Problems like there being not even a single clearly production-ready host platform out of the box. Centos? Not yet. Ubuntu? Best of the bunch, but still hacky and buggy. CoreOS? I thought one of the points was a unified platform for dev and prod.
The benefit of "containers" is that you don't need to siphon off a dedicated section of RAM to the application.
Nearly every time I install actual software on my mac (beyond editors & a few other things) I feel like I end up tripping over it later when I find half my work wants version N and the other wants version M
Yeah, I think a lot of it is better resource utilization compared to VMs. At the same time, though, I don't think containers are the thing, but just a thing that paves the way for something very powerful: datacenter-level operating systems.
In 2010, Zaharia et al. presented [1], which basically made the argument that increasing scale of deployments and variety of distributed applications means that we need better deployment primitives than just at the machine level. On the topic of virtualization, it observed:
> The largest datacenter operators, including Google, Microsoft, and Yahoo!, do not appear to use virtualization due to concerns about overhead. However, as virtualization overhead goes down, it is natural to ask whether virtualization could simplify scheduling.
But what they didn't know was that Google has been using containers for a long time. [2] They're deployed with Borg, an internal cluster scheduler (probably better known as the predecessor to the open-source Kubernetes), which essentially serves exactly as an operating system for datacenters that Zaharia et al. described. When you think about it that way, a container is better thought of not as a thinner VM, but as a thicker process.
> Because well-designed containers and container images are scoped to a single application, managing containers means managing applications rather than machines.
In the open-source world, we now have projects like Kubernetes and Mesos. They're not mature enough yet, but they're on the way.
[1] https://cs.stanford.edu/~matei/papers/2011/hotcloud_datacent...
This exploits the fact that one webserver only differs from another by the contents of it's response method, and other differences are actually unwanted. You can make this a lot more efficient by simply having everything except the contents of the response method be shared between different customers.
This meant that all the Apache mod_x (famously mod_php and mod_perl) can manage websites on behalf of large amounts of customers on extremely limited hardware.
It does provide for a challenging security environment. That can be improved when starting from scratch though.
Mod_php is 3 syscalls and a function call, and can be less if the cache is warm. Despite the claims on that page, there is no comparison in performance.
"Extremely efficient use of resources"
It is utterly baffling that one would use those words to describe spinning up either a container or a VM to run these lines of code (their example), and nothing else:
p := &Person{Name: "World"}
json.NewDecoder(os.Stdin).Decode(p)
fmt.Printf("Hello %v!", p.Name)
So I guess my issue is that functions use $massive_ton_of_resources (obviously the lines of code printed above here need their own private linker loaded in memory, don't you agree ? It's not even used for the statically linked binaries, but it's there anyway. Running init scripts of a linux system from scratch ... yep ... I can see how that's completely necessary), but when they're not called for long enough, that goes to 0, at the cost of needing $even_more_massive_fuckton_of_resources the next time it's called.
Of course, for Amazon this is great. They're not paying for it, and taking a nice margin (apparently about 80%, according to some articles) when other people do pay for it.
And the really sick portion is that if you look at how you're supposed to develop these functions, what does one do ? Well you have this binary running "around" your app, that constantly checks if you've changed the source code. If you have, it kills your app (erasing any internal state it has, so it needs to tolerate that), and then restarts the app for the next request. Euhm ... what was the criticism of mod_perl/mod_php again ? Yes, that it did exactly that.
Take the example here: https://github.com/kstaken/dockerfile-examples/blob/master/n...
Which does something a lot of these functions will do : get nodejs, use it to run a function. Just the apt-get update, on my machine just those instructions, ignoring actually running the function (because it's insignificant) does close to 1e6 syscalls.
For example, on my FreeBSD boxes, I have runit services that are basically this:
exec jail -c path='/j/postgres' … command='/usr/local/bin/postgres'
Pretty much the same as directly running /usr/local/bin/postgres except the `jail` program will chroot and set a jail ID in the process table before exec()'ing postgres. No init scripts, no shells, nothing.
Does it have it's own filesystem that can migrate along with the program ? Does it have it's own IP that can stay the same if it's on another machine ?
In my example, /j/postgres is that filesystem that can migrate anywhere. (What's actually started is /j/postgres/usr/local/bin/postgres.) Yeah, you can just specify the IP address when starting it.
This is often done via a super thin distribution like Alpine Linux to keep image size down, despite the COW functionality touted by Docker that's supposed to make it cheap to share layers.
The difference is that unlike a fully virtualized system, the container does not have to execute a full boot/init process; it executes only the process you request from within the container's image. Of course, one could request a process that starts many subservient services within the container, though that is typically considered bad form.
What people really want is super cheap VMs, but they're fooling themselves into believing they want containers, and pretending that containers are a magic bullet with no tradeoffs. It's scary times.
> [the Apache/CGI model] does provide for a challenging security environment. That can be improved when starting from scratch though.
And the number of lines of code in the example probably doesn't quite matter so much, because that's all it is: an example. I am sure that you can run more lines of code than that.
> Euhm ... what was the criticism of mod_perl/mod_php again? Yes, that it did exactly that.
I mean, that's also basically my point, that Lambda is basically the CGI of the container world. Lambda and CGI scripts really do seem like they are basically the same thing; I still speculate that they will be used to fill similar use cases. I am not really opining on which one is actually better.
It is still safer than containers as one kernel local root bug does not break a VM, but breaks a container. The access to hardware support also allows compartmentalized drivers and hardware.
- have different versions of libs/apps on the same OS (or run different OS's) - tinker with linux kernel, etc without breaking your box (remember the 90's?) - building immutable images packed with dependencies, ready for deploy - testing distributed software without VMs (because containers are faster to run) - if you have a big box (say 64gb, eight core or whateva) or multiple big boxes, you can manage the box resources through containerization, which can be useful if you need to run different software. Say every team builds a container image, then you can deploy any image, do HA, Load balancing, etc. Ofc this use case is highly debatable
That being the case, why not just use the OS? And processes and shared libraries?
The other big advantage is containers provide a way to distribute and run applications with all their dependencies except for the kernel. This means not having to worry about incompatible libraries or installing the right software on each machine.
That's essentially what a Linux container is: a process (that can fork) with its own shlib. If you have lots of processes that don't need to be isolated from each other and can share the same shlib, then no, you don't need this mechanism.
And if you want to use a modern development tool chain you don't really have this choice. They produce statically linked binaries that need, at minimum, their own process and TCP port (if you run a proxy which when you think about it is pretty wasteful).
There is no good reason (other than ease of tool chain development) for that, and it's probably cost hundreds of millions or even billions of dollars in servers and power, but there you go.
PHP and Java are essentially the only languages with good support for running without containers, and Java isn't even used that way usually.
Sharing library code between processes running in containers is more complicated, since it depends on whether and how you've set up filesystem isolation for those processes, but it's possible to do.
The other big difference between this and a VM is that timekeeping just works.
You're not necessarily restricted to friendly-only tenants, either. Depending on how you configure it, there can be pretty good isolation between the inside and the outside and the other insides. You lose a layer of isolation, but it's not impossible to escape a virtual machine either.
There are companies doing multi-tenant Container setups, with untrusted customers, so it's not an unknown concept for sure.
what I'd say is that the attack surface is much larger than a VM hypervisor , so there's likely more risk of a container breakout than a VM one.
I'm a little shocked to hear this (given everything everybody else has said about container security), but I guess it means the security of containers can be tweaked to be good enough in this environment.
Examples?
Ended with a spectacular data loss, of my own company's financial data. Luckily I had 7-day old SQL exports.
1. More efficient use of hardware (including spin up time) 2. Better mechanisms for tying together and sharing resources across boundaries.
But in the end they don't really do anything you couldn't do with a VM. It's just that people realized that VMs are overkill for many use cases.
In fact, nobody guarantees that say Fedora will run on an Ubuntu-built kernel. Or even on a kernel from a different version of Fedora. So, IMO, anything other than running the exact same OS on host and in container is a hack.
"nobody guarantees" just means that you can't externalize the work of trying it and seeing if it works. I don't think that's a huge loss, considering the space of all possible kernels, configuration switches, patches and distro packages is huge.
It's like refusing to use a hammer because nobody can assure you that hammer A was thoroughly tested with nail type B.
As long as the ABI is stable and you don't reach out to something that would have moved within /{proc,sys,whatevs}, you're good. [0]
I'd say the "much higher" is nowadays a relic of the past.
Even if you understand them, you have to understand the specific configuration (unlike VMs, where you have a very limited set of configurable options, and the isolation guarantees are pretty much clear).
Going beyond this you get orchestration - which you can certainly do with VM's but it's slow; and various hangovers from SOA rebadged and called microservices.
But they're really, really efficient compared to VM's.
They are definitely not VMs.
> But they're really, really efficient compared to VM's.
I think that the virtualisation CPU overhead is below 1%. Layered file systems are possible with virtual machines as well so disk space usage could be comparable.
What do you mean that they are "really, really efficient" ?
Really really efficient relates to how many containers can be run on a given system vs VM's. About 10x as many.