Docker Development Patterns
hokstad.com
hokstad.com
The project should have a single Dockerfile from which dev/test/staging/production work from.
There are certaily things you may want to change in these envs (particularly dev and test) but these can be runtime configurations (through env vars or changing the command that gets run in the container).
When I do dev, if I need to do so interactively I build from my main dockerfile and do `docker run -v $(pwd):/opt/myapp myapp bash`, which mounts my live code into the container, overwriting the code that was built in. I'll mess around with some things, exit and adjust my Dockerfile if needed based on those changes.
To run tests (as an example): `docker run -e RAILS_ENV test -v $(pwd):/opt/myapp myapp rake test` If all is well and I'm ready to push my code up, go ahead and rebuild the image with the changed code, run the tests again, etc.
Sometimes it is neccessary to have some tools like gdb, strace, etc, available within the container. For this you can startup the container as you normally would, then (with Docker 1.3) `docker exec -it /bin/bash`, install the tools I need to debug something, and get out.
In general it is just going to be bad practice to have a significantly different from the production env, and is indeed why dev handoff to production is often wrought with trouble.
I don't agree with this largely, I think because we actually substantially agree about the need to test regularly in an environment identical to staging/production during development. I DO agree that there should be a single Dockerfile to be used as the basis, and I DO agree that you should regularly run that Dockerfile in dev/test/staging/production unmodified.
But part of the point is that there is no reason to limit yourself to a single dev instance of your app. I appreciate that might have been somewhat obscured because of some of my examples, which uses a separate dev container setup that I use for some projects as the base, including as a base for the production container for my blog.
So to reiterate, I absolutely agree with having a Dockerfile that is identical to your live environment to bring up an instance with during development.
But that is not a reason to not _also_ have a container that bring it up with a code reloader, or a container that lets you test in different environments. And it certainly is not a reason to avoid isolating dev/build dependencies into yet other containers.
I actually much prefer that to the "Rails way" of having separate environments "built into" the app, and that's probably part of the difference. Instead of doing that, I use "FROM" in Dockerfiles to "inherit" from bases to provide me with different "lenses" to work on the project from different viewpoints. Not just dev vs. live, but also with debugging tools, or architecture differences etc.
> Sometimes it is neccessary to have some tools like gdb, strace, etc, available within the container. For this you can startup the container as you normally would, then (with Docker 1.3) `docker exec -it /bin/bash`, install the tools I need to debug something, and get out.
But there's no reason to keep redoing these steps when they can be encapsulated in a Dockerfile that inherits from your basic container. And you can easily run it in parallel with your "pristine" containers, or bring it up in seconds as needed.
> In general it is just going to be bad practice to have a significantly different from the production env, and is indeed why dev handoff to production is often wrought with trouble.
We fully agree with this. It's a decade of frustration doing devops that has made me enjoy the opportunities Docker brings to layer situation-specific dependencies on top of a pristine base: It takes away any excuse for letting the basic setup diverge, or for being unable to run full tests on an environment identical to live.
Since volumes are bind-mounted from the outside, you can put the volumes on whatever storage pool you want that you can bring up on the host (as long as it meets your apps requirements, e.g. POSIX semantics, locking etc).
E.g. at work we have a private Docker repository that runs on top of GlusterFS volumes that are mounted on thehost and then bind mounted in as a volume in the container.
I also run my postgres instances inside Docker, with the data on persistent volumes.
We could certainly use better tools to manage it, though.
Another pattern I ought to have mentioned (that I've used myself) is to set up "empty" containers whose only purpose is to act as a storage volume for another container. I don't like that as much, mostly since I've not had as much long-term experience with how the layering Docker uses would impact it.
The exception to this would be, I guess, you could wedge in Docker containers for database server isolation or something, but my databases don't run on multi-tenant instances so there isn't a huge win to it. (I use RDS most of the time, let somebody else manage that problem.)
Which is why pretty much all advice regarding Docker is to use volumes, so that the persistent data is managed from outside the container, and the container itself can be discarded at will without affecting the data volume.
My preferred method is bind-mounted volumes from the host. They are not part of the containers, and the purpose is exactly to remove the need or persistent containers. A lot of the examples I gave in my article relies heavily on this.
This leaves the form of persistence up to the administrator. And of course that means you can do stupid things, or not. On my home server that means a mirrored pair of drives combined with regular snapshots to a third disk + nightly offsite backups. At work, we're increasingly using GlusterFS on top of RAID6, so we lose multiple drives per server, or whole servers before the cluster is in jeopardy (and even then we have regular offsite snapshot throughout the day + nightly backups).
If you are referring to the pattern of creating "empty" containers to act as storage volumes, then I sort-of agree with you, but mostly because of the maturity of Docker. After all nothing stops you from putting the docker storage itself on equally safeguarded storage. It's not really the risk of losing storage that makes me prefer stateless containers, but that separating state and data substantially reduces the data volume that needs to be secured (since we can spin up new stateless containers in seconds, we only really care about preventing loss of the persistent data volumes).
I respect GlusterFS, so it sounds reasonable in your own use case, but it still makes me intensely uncomfortable to have apps managing their own data. I try to build systems where each component does one thing well, and for the components that I think work well in a Docker container, data storage is then kind of out-of-scope. YMMV.
Also, docker doesn't add as much value for something like postgres that likely lives on it's own machine.
Then don't use data containers. I don't see much benefit from that either. The stuff we put on data volumes is stuff we want to manage the availability of very carefully, so I prefer more direct control.
And so when I use volumes it's always bind mounts from the host. Some of them are local disk, some of them are network filesystems.
We have some Gluster volumes that are exported from Docker containers that imports the raw storage via bind mounts from their respective hosts, for example, and then mounted on other hosts and bind-mounted into our other containers, just to make things convoluted - works great for high availability (I'm not recommending Gluster for Postgres, btw.; it "should" work with the right config, but I'd not dare without very, very extensive testing; nothing specific with Gluster, just generally terrified of databases on distributed filesystesms).
> for something like postgres that likely lives on it's own machine.
We usually colocate all our postgres instances with other stuff. There's generally a huge discrepancy between the most cost-effective amount of storage/iops, RAM and processing power if you're aiming for high density colocation, so it's far cheaper for us that way.
We have definitely used them at large scale in production for several years with no issues.
It may be presumptuous to say the referenced PR will be in 1.4, but that's certainly what I'm pushing for.
I don't want to care if the container storage is totally destroyed - I want to just re-create it from our registry on a different host. I keep threatening the devs I work with that I'll wipe the containers regularly, for a reason, and they're specifically and intentionally not backed up.
From what I hear, the idea of keeping the containers totally disposable is a key appeal of Docker for many (me included), so I would just not use any volume related functionality that co-mingles the data volumes other than in very special circumstances (e.g. lets say we had static datasets that we'd like to mix with an application in different configurations; but I don't have any actual, real-life use cases for that at the moment; basically I'd only do that if I then also could push those volumes into our registry)
That said, it is hard to re-use a volume right now if you removed the last container referencing that volume. The linked PR solves this issue.
You can mount these SSD's on the host and then use bind-mounts (docker run -v /host/path:/container/path) to get those into the container. Or you can add the devices to the containers directly ("docker run --device /dev/sdb", for example), but you'd need privilaged access to actually mount the device.
The referenced PR itself doesn't really make using things like specialized disks any easier, except you can register them with the volumes subsystem and it will track them form you, ie `docker volumes create --path /path/to/data --name my_speedy_disks`.
There was some discussion around being able to deal with devices directly with Docker instead of expecting the admin to handle mounting those devices onto the host so they can be used as volumes. Nothing finalized here. Here is where that discussion happened: https://botbot.me/freenode/docker-dev/2014-10-21/?msg=239115...
For any package which we need to build, we have many packages with dependencies between them, are specified in a Makefile - because make is good at dependency tracking, but when make needs to (re)-build a package, it does so using the docker build-container-base image, whatever that package does is committed for later analysis under the name of the package built and timestamp, for traceability.
For every package built, the resulting docker image, with build-environment-and-package-plus-timestamp name, is run but now, trying to install the package and commit that image as well with 'install' tag. - so we also verify the install step basically.
Btw, this is for openstack, many components, packages and dependencies all over the place.
We use packer for a lot of other things though and it's great.