Docker Part One: Containers
theregister.co.uk
theregister.co.uk
Everytime I ask that, I get the answer "When you want to run applications seperated from each other in their own environment". Yeah, I get that. But what is the real world use case for that? Who wants to do that? Now I will get answers like "Everybody who wants to run applications in containers" :) Yeah ok. But who? Do bloggers need Docker? People who run onlineshops? Would Ycombinator benefit from running Hacker News inside a Docker container?
the same way you separate you model from your view from your controller, you can separate your various services from one another.
Your infrastructure consists of 5+ virtual servers that host your services. Each virtual server might be database only or db + dns or web + dns or ... you got the point.
What docker allows you to do is to isolate all those 20 services in separate containers that if any of them is being hacked your attacker will end up compromising only that service without any access to the main server.
What docker allows you to do is to version control the configuration and the container for the different services in a way that if you need to reinstall a service ( redeploy your app ) it is less of a hassle and since it is containeraized you will not be compromising your other services while doing that.
etc.
As for why Docker? Dunno. While I think illumos zones are pretty awesome myself, having a standardised format is handy. Docker is also popularising the container mantra with Linux.
In the scenario you're describing where you're provisioning virtual machines in a cloud, shuffling docker containers around on a fleet of VMs may not be the way to go. Many people would use some combination of disk images and configuration management to achieve the same ends.
It seems like the kind of thing that makes sense if you have a big deployment anyway, so you might as well hive stuff off, but makes much less sense if you're a "small" shop.
So when you are performing patching, you'd bring a container into service with the updated patches, send some traffic to it (possibly tightly controlled to minimise disruption), verify and then put into production. Iterate through your N-nodes per tier and you've patched. Every container after the first is rapidly started up (just the cost of a supervisor/init & the application process).
Have the Docker developers changed their stance that they don't promise that Docker can safely untrusted programs running as root? (https://news.ycombinator.com/item?id=7910117)
From the security section [1] :
> Your average server (bare metal or virtual machine) needs to run a bunch of processes as root. Those typically include SSH, cron, syslogd; hardware management tools (e.g., load modules), network configuration tools (e.g., to handle DHCP, WPA, or VPNs), and much more. A container is very different, because almost all of those tasks are handled by the infrastructure around the container ...
> This means that in most cases, containers will not need "real" root privileges at all. And therefore, containers can run with a reduced capability set; meaning that "root" within a container has much less privileges than the real "root". For instance, it is possible to ...
> This means that even if an intruder manages to escalate to root within a container, it will be much harder to do serious damage, or to escalate to the host ...
Except that: containers share the same kernel. If a contained application is hijacked with a privilege escalation vulnerability, all running containers and the host are compromised. Also, several resources are not namespaced. Nothing with a shared kernel is going to be very secure.
Malicious container can easily compromise a vulnerable host OS, and an already compromised host OS completely owns the container. This is no different from the failed experiments in sandboxing on PCs.
Can your container share a Linux instance with containers from other customers in the cloud? Not with any confidence, no. Containers do not contain: http://opensource.com/business/14/7/docker-security-selinux
Somehow at the same time I also messed up the Oracle .net drivers.. I am forced to use a specific version that the rest of my team doesn't use.. God knows how, but I have spent days trying to work out what has gone wrong. I am really looking forward to Microsoft implementing Docker for Windows. Then I will be able to develop each project in isolation and if something messes up like this again - it won't matter.
The problem I have with it is the level that it is done at. Using OS level virtualization (lxc) seems somewhat messy, because now you have to have a whole addition layer of proxies and indirection for any networked services.
I am left wondering if sandboxing/Capsicum[1], along with something like Nix[2], would be much cleaner abstractions in the long run.
[1]: http://www.cl.cam.ac.uk/research/security/capsicum/freebsd.h...
Nonsense. You can have LXC containers work at directly at ethernet-level with each container having it's own regular network IP, just like a regular VM.[1]
Just because the default is to create an internal network doesn't mean LXC is limited to this only.
And then all the problems you mention go away. You get the hygenic benfits of a VM without the resource-related cost.
[1] Simple example on setting this up on Debian: https://wiki.debian.org/LXC/SimpleBridge
I have a handful of VPS's. Every once in a while I find a fantastic deal on one and add it to my "collection."
Instead of trying to manage which VPS has what running on it - I use docker. Or rather I am migrating to it, still not 100% complete.
It is quite powerful in this regard, allowing me to write small separate applications that run completely agnostic from their underlying hardware/virtualization.
If I cancel a VPS, I just move the container over with a couple simple commands.
So how is this useful? This morning I decided I wanted to have access to a HTTP-DNS tunnel. I launched iodine in a docker container and pushed it to one of my servers - if I need to move it to a different one, it's just a couple commands away from happening without having to reset or rebuild anything.
I have a handful of web scrapers, RSS monitors, an IRC bouncer, etc running in docker containers that make management much easier than my previous solutions.
Now as for a more "professional" use? I'm not as sold on it. If your server has room for multiple "things" running on it - you a.) need multiple smaller servers or b.) your application isn't large enough to even need docker to begin with. But of course this isn't applicable to everything and docker definitely has other benefits that depending on your software may or may not be useful.
1.) Packaging and configuration development. I find it nice for this purpose because a.) it makes it easier to ensure that the work I do does not depend on some quirk of my local system configuration, which has evolved over a long period of time, b.) it is easy to tear things down and start over without consequences, c.) the versioning/layering can be useful for evolving base images
2.) Encapsulating research work and working environments. I can build a full, working environment for my research, with all source and compiled binaries, and be certain that everyone else who comes along will be able to use it and replicate/reproduce my work, or extend it with their own, without a need to hassle with configurations, compilers, etc. I can index the container with tags from a paper or conference so it can be readily found and used.
I guess none of these things requires Docker [or any lxc system for that matter], but I have found it really useful and easy-to-use for these two purposes.
1) Devs get to use whatever environment they want, but the product they work with is guaranteed to be the product on the servers - no mismatched libs and a reduction in WORKSFORME.
2) For incorrigible hackers who like a finger in every pie. Want to work on project X that depends on a given package? Spin up the container. Want to shift to a different project that uses an incompatible version of that package? Spin up a different container.
I'm not seeing it live up to the hype in my ops world - to me, it's just a package-management system with it's own baroque API and a custom init system. Then there's things like the python bindings using a different workflow to the CLI (the python bindings put config in 'start', the cli cannot do this). Cryptic cli responses. The 'docker ps' output that has crazy long whitespace and I have yet to see not wrap to become an unreadable mess on any of my terminals. Downloading a new image not only takes the 'latest' tag from the previous (fair enough), but it also steals the 'repository' name - so now the previous image has neither the tag (fine) or the repo name (!?!?!). I'm also yet to figure out how to get dockerhub to send an email that the build failed. And fun stuff like the benefit of adding yourself to the 'docker' group so you can run docker commands without having root... but you can forget about looking at the logs, no access there!
Yesterday I learned that they also intend to remove usability features rather than add them, with the belief that 'third-party apps' will provide them. Because fragmentation is good when it comes to usability? The functionality is already there, and they want to remove it to 'keep the core small'... rather than perhaps move it into an 'extras' repo (that everyone will end up adding anyway, bar the die-hard minimalists). https://github.com/docker/docker/pull/5001
I'm finding that I'm fighting docker more than it's helping - I am yet to personally see the advantage to ops over a .deb package. Or an rpm, if that floats your boat. But then again, I'm already using largely single-purpose VMs. I guess you can store state in a docker image that's not as simple in a .deb, but then again docker linux images (based off distros) generally start at 100MB and grow quickly from there.
And to make matters worse, my developers who went "ooh, shiny, we HAVE to use Docker!" and pushed the issue... aren't even using it themselves to develop our product. They grew bored of it quickly and reverted to their usual methods. I get puzzled stares when I run into the docker-corner-case-of-the-day. So much for item 1) for me...
/rant
docker.errors.APIError: 404 Client Error: Not Found ("client and server don't have same version (client : 1.16, server: 1.15)")
In practice, the main use case for many companies is to use docker to package Ruby, Node or Python applications along with all of their module and OS library dependencies. This allows multiple applications to run on the same physical machine without conflicting with each other.
I have only occasionally had Python scripts broke because of dependencies, but not since the introduction of virtualenv.
I don't even know how to process a comment like that about Virtuozzo. The fragility of Virtuozzo and OpenVZ is infamous. We get more support queries about our software running on OpenVZ and Virtuozzo containers than all other virtualization types combined (our stuff runs on a few hundred thousand hosting servers, we have a really good sample size).
For years, whenever we get a mysterious "this thing stops for no reason" or "this error doesn't make any sense" or "sometimes it works and sometimes it doesn't" support request, literally our first clarifying question is, "Is this an OpenVZ or Virtuozzo system?" It is, frankly, broken by design, with regard to memory allocation.
Well, it'd help if you offered a counterexample to the original claim refuting it.
If intermittent failures without useful error messages is the worst kind of failure (and it is, IMHO), then Virtuozzo intentionally subjects its users to the worst kind of failure. Unless you know how to spot this kind of problem, it's easy to bang your head on it for days or weeks; and we've had users who were ecstatic when we diagnose their problem and provide the knowledge they need to spot it in the future (though resolving it may be more tricky than diagnosing it). Even without the host intentionally overselling, we've seen this behavior happen, because allocation seems to require contiguous blocks of memory (or something else that means that even with enough available memory allocations can fail).
I am not deeply familiar with the code, only with spotting this problem in the wild. I've seen assertions, spanning years (including from Parallels), that it's been fixed, or that as long as the host isn't overselling RAM it won't happen. But, we still get these support requests regularly. I'm not an expert on their software, but I'm an expert on spotting when it's their software that is causing our customer's despair. (Note that the support requests are usually about Postfix, Apache, MySQL or other services "crashing"; it rarely manifests with our software specifically, since our software rarely needs to allocate large amounts of memory.)
If you happen to be stuck on a Virtuozzo or OpenVZ system, and have mysterious failures of services, you can find out of it is this problem by checking /proc/bc/* or /proc/user_beancounters.
How else would you suggest that OpenVZ should handle over-allocation? It is, after all, one of the points of virtualization (and the reason why you can get such containers for cheap).
You can't really blame OpenVZ if the users or sysadmins are unaware that the container has hard limits and that you're able to see if you've run into them using /proc/user_beancounters. Of course, if OpenVZ fails to allocate memory when you're under the limit, that's a separate problem that should be adressed (although I've yet to run into that one).
Otherwise I agree with you that admins should detect VZ-related OOM conditions. But a good project takes ownership over the most common pitfalls and finds ways of making their software easier to use (if their goal is to remain popular or grow).
It isn't my problem, it is Parallels problem, and the problem of their customers. Though, it becomes my problem when our customers come to us for help due to Parallels failure.
You say it is working as designed, I say that design is broken. Those are not mutually exclusive assertions.
"You can't really blame OpenVZ if the users or sysadmins are unaware that the container has hard limits and that you're able to see if you've run into them using /proc/user_beancounters."
It is not the "hard" limit that is the problem...it is the "soft limit" which is arbitrary and can be unallocate-able, with the user having no way to know that it will be unallocate-able until it causes a service to exit. So-called "burst" RAM in VZ containers would better be called Schrödinger's RAM, and it has no place in a server.
Further, it doesn't even seem to exclusively be a problem of "burst" RAM. We've had support requests show up on systems that had, say, 1 GB of "guaranteed" RAM and 2 GB of "burst" RAM, but still behave as though it has significantly less than 1 GB of allocate-able RAM. I've addressed that in my comment above, but I didn't give an example. Perhaps this is more clear?
Regardless, the behavior of VZ containers even when they are working as designed is so user-hostile that I just can't recommend it. Our products support OpenVZ containers, because we had a number of customers tied to the platform from before their were better alternatives, but I've never been fond of them.
If it works for you, and you don't share your system with anyone else, go ahead and do what makes you happy. But, from my perspective, as someone supporting probably more web hosting server users than anyone else I know (except maybe some friends who work at large hosting providers and a friend who works at our largest direct competitor), I know that VZ containers cause a lot of confusion.
Been there, done that, run an illumos distro now; it's like night and day. Calling Virtuozzo the best container system is a bad joke. Same applies to OpenVZ.
Speaking of Docker, I did find some interesting repos today: https://github.com/socketplane/socketplane and https://github.com/ibuildthecloud/only-docker
Docker has a whole ecosystem of bolt-on github projects that fill-gaps that the Docker project hasn't had time themselves to fill because they are too busy organising conferences, withholding features until the keynote of said conference, and working out how to spend the millions of dollars on future development.
Even the gap-filler github projects are getting VC funding (see http://weaveblog.com/2014/12/03/announcing-weaveworks-and-ou...). Weave will either become the standard networking stack, or be made irrelevant by an in-tree featureful implementation.
So Docker are on this tightrope trying to innovate whilst avoiding stepping on the toes of their VC-funded 'partners', so their response was simple-core with a plugin framework for partners to integrate into. Anyone else picturing fragmentation in 6-12 months time?
Does part 4 finally deliver on the promise in the title?
You can always scale out with docker later and I know at least ansible are adressing this use case (http://www.ansible.com/docker).
Saltstack in particular has really good Docker integration at this point, which may be useful. YMMV.
I would avoid Docker until you have a use-case that Docker solves for you. Use your preferred distro's packaging technique. It can be a bit of a pain (.deb in particular) to start with, but once you're familiar, it's portable and manageable with your OS's native tools. Things 'just work', and don't need an agent to be installed and kept up-to-date. If something goes wrong, it's in your OS's standard logs, not some strange arcane location. And most importantly, you have a bigger pool of interneteers to help learn from and troubleshoot with. I can't complain about the Docker IRC channel, good experience there, but there's still fewer of them.
If you are a .debber and want your own repo, check out Aptly, which makes life much easier than previous repo software on offer. http://www.aptly.info/
TL;DR: Docker is 'another thing to learn', and it's not trivial. Put it off until it solves a problem for you; there's plenty of other stuff to get up to speed with.
Lets say you have a server, it runs a mail server, database server, web server (proxy) and application server.
If they all run without containers, if one service gets compromised, and a root exploit is found, that's it game over.
If you have a service that starts eating up memory, with proper configuration, it can't overload the root server. Basically each jail/container can only see itself and any exploits cannot effect the host system, or other jails/containers (when configured correctly).
It also allows for easy expansion, when one jailed/containered service gets to large for the server, just move it to another server easily and quickly.
It also allows for speedy deployment, with docker you can bundle everything on your laptop, create an image, then ship this straight to an external host like EC2/Goole Cloud (for example), with the addition of pre built containers for Django/Rails/Postgres/MySQL etc, it creates a ready working environment for developers who might not be to hot on configuring systems. The "shipping" ability to docker is a by product of the container which is another added benefit. There are loads more features than what I have stated here, this is just a very brief summary.
To be fair, if a kernel-level root exploit is found, it's probably also game over for containers. It's possible to have root exploits that cannot escape containers due to UID virtualization or whatever, but typically(?) root exploits are based on being able to mess with kernel memory, in which case escaping a container should also be possible.
For root exploits isn't Docker toast as well? I haven't followed Docker in much details, but does Docker actually promise that commands run as root will be contained?