Is Docker ready for production?
t37.net
t37.net
I'm expecting that to shift with both Docker and the ecosystem of tools and services around it maturing, to the point where in a few years time Docker may even be advantageous to the most simple setups.
I'm really excited about Docker, but personally I'm not expecting to have any use for it in production for another two years.
Deis is a fully opensource PaaS inspired by Heroku. You can dip your toe in the water by spinning up a Deis cluster and then just following the Heroku inspired/compatible workflow described in the Deis docs. When you are ready to experiment Deis let's you use Dockerfiles instead of a Heroku inspired workflow.
Once you have your Deis cluster up you might even want to sidestep it entirely and run Docker containers directly on your CoreOS cluster.
Deis can be used with Vagrant and VirtualBox and it can also deploy to multiple clouds (EC2, Rackspace, Digital Ocean, Bare Metal (e.g. Kimsufi should be possible http://bit.ly/1t2PPXB). I've been playing around with it and it looks great so far.
It seems a too heavyweight solution (Deis cluster has its own instances of PostgreSQL, Redis, a Docker image registry etc) for the 'humble web developer' scenario.
Plus it's a bit worrying to use Heroku buildpacks on something that's not Heroku... you're going to be relying on the Heroku docs and then having to filter out details that aren't relevant or incompatible
Are we doing it wrong? Should everything be Docker and all of our Chef recipes should be written for Docker? Or is this right and just a natural pain of having both bare metal processes and Dockerized processes? Unlike the article, so far everything is 1 process or application per container, not a full OS.
Docker pretty much removes the the need for such complexity. Don't think of it as a provisioning tool, like chef. Don't think of it as a VM. Think of it as an isolated filesystem and process environment that eliminates the complexity of supporting multiple, disparate application stacks.
The storage for persistent storage with Docker is less compelling, because now you have shared resources between containers that should be isolated. That said, we run Cassandra clusters in Docker and have no trouble.
Also, If you want to re-use the work you've done with Chef, there are chef plugins for Docker which make it easy to provision Docker images with chef (solo, I think)
In the case of Cassandra I'd have certain cassandra.yml tweaks I need to make, possibly different for beta/staging/prod, and docker-cassandra needs that config file. I also need to spawn the containers with the right setup, such that the right ports get exposed and shared volumes get mounted so the configs get read in. The spawning is a slightly different ops problem (do I want a homogenous cluster where I just spawn containers somewhere as needed, or do I want certain containers on certain machine types), but it seems like something that would fall in the domain of a provisioning tool.
Should I just be building the configs into the container and rebuild the container if they need to change? The isolation of the process seems at odds with my desire to have a semi-dynamic, centrally managed configuration.
I'm also considering that Chef itself is overkill to manage Docker containers and there are other tools more well suited to managing Docker-based infrastructure, but I'm not sure what they are.
We've been using chef in the past to manage docker, which never felt very good because the main cookbook was somewhat buggy (but it has gotten better recently).
We now use mesos+marathon(+chronos for deployment) for running our docker container, which sounds much more overkill than chef, but turned out to work much better and cleaner.
I'm using (b) for our nginx configs, you mount only the single config file as read-only, then let chef copy up the config before mounting/launching.
https://github.com/kelseyhightower/confd
Basically, you add confd into your Docker image and execute it at runtime to do just-in-time config file generation. Here's an example for nginx:
https://github.com/grosskur/nginx-confd-dockerfile
You can create separate environment variable files for beta/staging/prod and pass --env-file to "docker run".
This lets you use the same Docker image across all your environments and avoid the operational complexity of mounting config files in Docker volumes.
One approach might be to base the final Docker image on another Docker image, which has a snapshot of all Rubygem dependencies at a certain point. In the depending image, the 'bundle install' will then do an incremental update and the Docker build will go a lot faster.
But I was wondering how other people are solving this?
This means I can run multiple rails apps using the same image just by changing what volume to mount
I still prefer to deploy my Ruby apps as part of any image, though. Of course you can combine the two: Build an image whose sole purpose is to package your app and export it as a volume for use by pre-made Rails/Nginx containers.
A typical large Ruby app ends up dragging in tons of build dependencies that does not need to be in the final image to be able to do things like compile extensions etc.
# This only re-runs the slow installation when requirements.txt changes
ADD requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt
# This re-runs every time you change any file, but is very fast
ADD . /app/http://ilikestuffblog.com/2014/01/06/how-to-skip-bundle-inst...
In one you install deps, and in the other, which relies on the first, you install your app. If you change your deps, then you rebuild the first.
I'm not a Ruby dev, but this seems like it should be quite simple?
I think OP wants to not have to reinstall every dependency again when they update just one. For that I think you'd have to use multiple Gemfiles, one for slow things that rarely change and the other for anything else new.
> The final image is 570MB big. I could not shrink it more unless I remove the whole Python and Perl stack. Since both are necessary for many system dependencies, starting with apt-get, this was not possible. I still need a way I can improve or upgrade my container.
?!? The article starts by pointing out they use immutable servers and blue/green deployment. In that context, you will not improve or upgrade the container: You build a new one. And if you want to cut build dependencies from the final container: Do the build in one container, install the build-artefacts to a volume, and use the contents of that volume to build a container without the build dependencies.
It'd be great to get "built in" support for this, but it's not hard to do.
> There’s no easy way logging with Docker.
The standard way of logging with Docker is to log to standard out, which gets captures and is accessible via "docker logs". If he'd not dismissed systemd out of hand, he'd also easily have gotten it fed into journald, with the option of having it relayed to a remote or local syslog as per his preferences.
> Let’s put it this way: as a way of provisioning a container, Dockerfile is a joke.
We don't need more complex provisioning tools. We have plenty of provisioning tools. Ultimately Dockerfiles needs to be able to specify what should be copied into the image. Everything else you can do with your standard/preferred build tools. There's no reason for Dockerfiles to try to become yet another fully featured provisioning tool.
> Forget your classic monitoring (unless you want to pull your hair with network bridges). Everything you’ll be able to monitor within the container are ports. That because you run the old school nrpe inside your host, so you won’t be able to check you actually have 8 workers running inside your container.
This is just flat out wrong. Anything running on the host can see the processes running in the container. With the right cgroup manipulation (via nsenter etc.) it can also see the mounted volumes or network space of a container, and so you can still monitor whatever you like.
> Making your application Docker compliant requires you to rethink the way it works.
Making your application take advantage of Docker, rather than treating Docker containers as sort-of VM's with less isolation requires you to rething the way it works. It's not something you need to do in one go - you can "break apart" a larger app environment piece by piece.
> The the tag nightmare begins. If I update my application and add new deps, I’ll have to update container #2. Unfortunately, how will I know I have to do that?
Uh. How does he know he has to update the machine images he deploys his applications to today? Personally I use make - tracking build dependencies is what it is for.
For size, it appears the OP started on his static compilation quest basing his image on Ubuntu. Wheezy is the standard base image in the Docker world for a reason -- it's significantly smaller. More and more images are using busybox based images, but I wouldn't want to try that with Rails + ImageMagick.
As for logging, I'd like to point out logspout: https://github.com/progrium/logspout
Tried building a piece of infrastructure with Docker quite some time ago and left when it just didn't click. I certainly did quite some mistakes (at that time I did try to stuff all the provisioning into the Dockerfile for example, which you agree is a bad idea?) and the official Dockerfiles were .. mixed in clarity and not quite useful as examples to me.
So, I would really love reading about how people provision and manage their Docker instances The Right Way.
I'll be posting a more detailed walkthrough of my process for using Docker with Ruby soon, as well as some other experiences.
> (at that time I did try to stuff all the provisioning into the Dockerfile for example, which you agree is a bad idea?
It's not necessarily a big deal for everything. I you have a lot of stuff that have limited build dependencies beyond Debian's "build-essential" target, you might as well create a debian:wheezy image with "build-essential" installed and work from that to begin with.
The issue is that some types of apps, like any Ruby app that uses Bundler and has dependencies on stuff with native extensions, can drag in hundreds of MB of dependencies that are only needed during build, and the build is only necessary when adding dependencies. For those kinds of apps it just wastes time and bloats your images to do the builds in the same Dockerfiles that you deploy with. If you're deploying 4-5 images now on a server or two, that's no big deal. When you're deploying 500 containers, you start getting picky about anything that slows down the process...
For those situations, I'd typically put together a Dockerfile with all the basic dependencies, and then "inherit" from that once for a dev/build container with the build dependencies, but without an entrypoint (so I can easily start it into bash or have it run a build script, or whatever I want), and then secondly for the final container where the latter image just adds the build artefacts.
Ideally I'd want everything I deploy to just have the files that are needed. Pragmatically, the tools are not quite there yet. Not least, it's often a real pain to build apps statically (if it's even possible), and the known dependencies are too granular (e.g. dependency on a large package, with no documentation about precisely what in the package has to be present) and it will take a long time to build up a "catalog" of more minimal images and build processes.
Absolutely.
Thankfully, the Docker team seems in agreement with this based statements about avoiding making Dockerfiles "too clever" and the response to various proposals.
As you point out, most of the "issues" here are really misconceptions.
I expect it's a tough balance for any new(er) project. Maximizing exposure and adoption, but avoiding negative perceptions from being applied in ways aren't optimal.
My experience (over the last year) is that they're so limited as to be pretty useless. They don't even do what they're advertised to do, ie give you a reliable way to reproduce a build, and they're inflexible for my idea of real-world work with Docker. Where they're good is in giving everyone a point of reference.
I had a discussion with the maintainers last year about this:
https://groups.google.com/forum/#!topic/docker-user/3pcVXU4h...
I have a problem with most CM tools in that they're for moving target systems, not immutable ones. Ansible is the closest, but our experience has been that development on it is slow relative to the tool we use (see below). It's saved us a ton of money.
I blog on this and similar topics here:
http://zwischenzugs.wordpress.com/
The "tool for building and maintaining complex Docker deployments" is here:
http://ianmiell.github.io/shutit/ https://github.com/ianmiell/shutit
I also talk about this here:
Not exactly, as the thread you link points out you can reference an image ID in FROM rather than the name:tag which has potential to change silently.
It's the equivalent of using a package manager against a repo you don't own without pinning - expect problems.
This can be mitigated by FROM'ing via ID or avoided entirely by running your registry where tags are reliable.
Admittedly, these things are not necessarily obvious, but I think it's a bit disingenuous to paint Dockerfiles as worthless or broken.
That said, ShutIt looks very cool and seems to address exactly some of my concerns / desires about working with Docker.
I just don't agree with framing it in opposition to and at the expense of what exists.
There's value in a container description that is fully self-contained, transferable and 'dumb' enough to be transparent.
Having done _lots_ of builds lately I can vouch for that (see my blog for some examples).
In the end the image ID _is_ useful, but the dockerfile itself has limitations.
I agree with your last point as well - my evangelism comes from solving problems at my company in this way (which I know are not uncommon problems) rather than any belief that it beats others objectively.
> Porting your application to Docker increases complexity. Really.
I think the main problem of Docker is that it's sold as an 'easy solution' by many bloggers who only deal with it superficially and then move on to the next big thing. There are a lot of gotchas with docker containers and the creation of clean docker images that are not immediately clear when you start out. A lot of your standard Linux know-how is not applicable.
edit: Also, there are obvious security issues that are not immediately clear to most beginners, most certainly not from the tutorials.
One of my favorites: If you provision your database container with environment variables to create a dba user, and then link your db container to your app container, voilà, your app container will now most certainly have the dba login and password inside its environment variables: https://github.com/docker/docker/issues/5169
Complexity management is not a Docker-specific thing, but Docker makes it easier than before, and standardizes how it's done. I think that's the main added value.
That's why it's so frustrating to see developers making superficial forays into Docker and then declare it to be too complex. Yes, the simple and largely irrelevant stuff does get a bit more complex and you have to do some learning (and re-learning) before you use it for a production workload. But that's a trade off that a lot of us are willing to make to make the crazy-hard stuff significantly easier.
We're developers. Our tools should not be optimized for first use.
That goes both ways. It's equally frustrating to see developers making superficial forays into it and declaring it to be the magic bullet that makes everything simple.
The basic fact is that building and deploying complex software, managing dependencies, handling discovery - these are all complex things. There is no solution that makes it simple because it is inherently not a simple process.
Instead, we can only shuffle the complexity to places we're more comfortable in managing. For some use cases that's a dockerfile. For others, it's chef cookbooks [or other CM solution]. For yet others, it's both.
Getting your logs out of docker is a PITA, though. Your best bet is to use syslog and configure each application to send their logs to a syslog server. It's a consistent widely supported way of shipping your logs around. Relying on stdout logs isn't always enough. Many applications do understand syslog out of the box but do not necessarily send important messages to standard out. Dockers management of stdout logs shouldn't be relied on at this stage.
Logspout does look interesting and I wish I knew about this a few months ago but see my above comment on stdout.
This is similar to what Netflix, one of the early big proponents of immutable servers, does. They build a deb package, then they create an AMI with that package installed.
http://techblog.netflix.com/2013/03/ami-creation-with-aminat...
What I read: Thank you! I've been looking for an excuse not to like docker for a while and this gives me great ammunition for my argument against it.
I read the GP as 'I was skeptical, but looking at the list of rough points in the article might make me to give it a try myself' - as a kind of 'well, if _those_ are the bad parts it might actually be interesting after all'.
Failure on my side?
Docker is a layer on top of that - it's what prepares the file contents of the jails and looks after them while they're running. I think Docker can even manage FreeBSD jails, although i'm not certain about that.
It's not the production-readiness of Linux containers that is in question here; they're fine, although nowhere near as mature as jails. The doubt is about docker, the layer on top. If you wanted to make a comparison to jails, it would be to whatever the equivalent of Docker is in the jail ecosystem. I'm not sure what that is; either there isn't one, or it's an ad-hoc pile of site-local shell scripts.
I think the point of doing it that way rather than just running Ansible etc on all your containers is that it means you're running N copies of one master image, rather than running N images which you hope are the same because they've been configured the same way. This doesn't seem like a colossal win to me, but some people seem to like it.
Why it's docker that runs Ansible etc, rather than the other way round, i don't know. It's a bit like how rpmbuild runs the build tools that make the contents of RPMs, i suppose. But i really hate rpmbuild so i don't find that analogy very encouraging.
As something of an aside, i get the impression that Docker is most popular with people, or organisations, or people in organisations where for whatever reason developers don't have a lot of control over the configuration of the machines their software run on. That could be traditional inflexible siloed organisations, or small, flexible organisations which just don't have an existing investment in infrastructure automation or the resources to make one. In those situations, Docker gives developers a way to make control the environment around their software without having to configure machines. This strikes me as a workaround to a problem rather than a solution to one, though.
What I see as a problem here for many people is that they don't want to do this all themselves and maybe docker offers conveniences out of the box. I get that. Someone needs to work on the jails tooling to make it more approachable so people aren't inventing the wheel.
Are there others? Is anyone clearly winning in this space?
As it stands, it seems far easier to just create a monolithic container with a bunch of running processes and a supervisord than to break up the pieces and then have a more complicated deployment.
> As I’m looking for a way to build a container without having it host the whole build environment (such as Puppet modules, ssh private keys to the Git repository, etc)
Why would you need to check out something from git within the container? That sounds like an unusual setup. Normally you'd check out your repo, then run a build from within it.
> A Dockerfile to build a container with a basic Ruby stack.
> A Dockerfile from #1 to build the deps: checkout the application on Github, install the packages, run bundle install, then remove the application. I’ll be able to share this container with both applications. Big win!
> A Dockerfile from #2 to download the application from Github, then setup everything. So my dependencies are already installed, and it goes fast every time I don’t need to update them.
Shouldn't this just be
Application with a dockerfile like this
ADD Gemfile
RUN bundle install
ADD .
Run docker build.
That way it's all cached, and you don't repeatedly build the dependencies. Has the author not seen docker caching?
What do you mean by "shared volume" -- docker's "-v" to bind-mount the socket into the container? How else would you like to expose the socket to the container? Why is it a no go?
I've read that using Debian (or if possible Busybox) helps in this area. See http://container-solutions.com/2014/11/6-dockerfile-tips-off... and http://jonathan.bergknoff.com/journal/building-good-docker-i... for tips.
> Every time your application is updated, check the Gemfile md5 and see if it’s different from the latest build
Wouldn't you need to check Gemfile.lock? With optimistic/pessimistic version constraints you could get newer gems without updating the Gemfile.
Exactly. It should be Gemfile.lock that is checked, not Gemfile