Experimenting with CoreOS, CloudFormation, confd, etcd, and fleet

[1]: http://docs.docker.io/installation/ubuntulinux/

  [Service]
  EnvironmentFile=/etc/environment

If you were trying to build wordpress-as-a-service, for example, you might create a Docker instance which includes nginx, php-fpm, memcached, and maybe even mysql (which makes it easy to make snapshots of the entire state of the system).

That said, fleet has a really interesting way of making interconnected systems, so you can create a memcached instance, an nginx instance, and a php-fpm-running-wordpress instance, and you can manage them separately. This makes it easy to upgrade memcached without touching nginx or php-fpm, or to change the nginx config without having to affect the running application server (and risk flushing the APC cache), etc.

It also allows you to make everything pluggable in your design. You might want to add varnish in front of your nginx instance, for example, but sometimes you might not. That sort of flexibility is really useful in some circumstances.

In our infrastructure at work, the design I would use would probably be similar to this; we have front-end services (nginx), internal services that they proxy to, internal services that those services access, asynchronous processing services, dispatch services, activemq, redis, memcached, mysql, etc. Having the flexibility to add more of any one of those things without having to scale any of the rest is really important.

For other circumstances, it makes a lot more sense to bundle everything into a faux-VM. Run supervisord to launch ssh, nginx, php-fpm, memcached, and whatever else needs running. Share volumes to local filesystems so that configs, local changes, etc. can be taken care of, and you're good to go.

So in the end, the real question is what do you want out of your containers? Flexibility or simplicity?

However, in a real life application I think I would still keep the containers as "single-process" as possible; in our case that would mean: a single Rails application or a group of Resque workers. It would make the containers quick to boot and would increase security by sandboxing all the separate parts of a system in their own containers. I do think this is up to personal preferences, though :-).

Right now, it is so easy that I even have (without CoreOS) a Hubot in a container, with its Redis brain in another container. Hubot can talk to Redis just fine, and that's the only container in that machine which can do so.

Maintaining the data storage backend gets easier this way. The Redis data is stored in a volume.

Mesos has preliminary docker support, and it's reportedly improving in 0.19; see https://github.com/mesosphere/mesos-docker

It also has primitives built in to get you thinking about fleets of machines instead of individual machines such as a default shared configuration and low level knowledge of the other machines in the fleet.

Lastly, since it is very Docker friendly, it gets you thinking in containers, which is a very neat way of making sure that your software is consistent across machines.

the hardest part is always shedding the baked-in habits.