Docker containers should not run an SSH server

> What is the general pattern of usage with Docker containers? Are they supposed to isolate just one application and do IO via a networks socket?

The maintainers of Docker appear to strongly believe that 1 application per container is the way to go.

At sufficient scale, they are pretty much correct. You grab Y bare metal hosts and spin up X docker containers for X processes.

> Isn't this just pushing the problems into managing dependencies between more little VMs while also constraining the architecture? It increases the difficulty of synchronizing 10 or 20 separate isolated applications start,upgrade,fail. Maybe all that living on yet another VM guest machine (so conceptually having 2 virtualization levels). Handling more complicated network setup (bridging, firewall rules at multiple levels). Handling effects of disk and other subsystem interacting with each other in strange, sometimes sub-optimal ways.

A handful of people advocate for immutable role-based containers [e.g. https://devopsu.com/blog/docker-misconceptions/] for that reason. In that use case, its really replacing running something like Xen + Chef, KVM + Ansible, or whatever.

You grab a host machine, you stand up your X containers with Z processes per container on Y hosts.

I think this is really equivalent to the Microservices vs. Normal SOA vs. Monolithic argument. Given sufficient scale/requirements, each makes sense. However, none of them are optimal for all situations.

I think that's just an issue of tooling and getting used to a different way of working with the services. If you run each of those services as a single docker container and group them on the same server, there does not need to be any more service discovery: You can link all of them. That still gives you the flexibility of modifying the linking without having to modify any of the containers.

The main practical difference vs. running them all in one vm is explicitly documenting dependencies - both which service relies on which package sets etc. (assuming you're strict about what you put in the Dockerfiles) and which part of your system needs access to which other component or needs to share access to which directories.

You can also easily start them e.g. via systemd unit files or via a tool like fleet, and pull the ip/port of dependencies on startup of the container and pass them as arguments. This also makes deploying them as a unit easy, but still decouples it:

If you need to move one component to a different server, or suddenly realize your foobar server needs lots of resources and want to load-balance it across multiple machines, you can easily do so by forwarding a port via stunnel or load-balancing via haproxy or what have you without affecting the actual containers.

As for your health checks, you can still have a single all-encompassing health check if you want - whether it's in one "vm" or a set of separate containers does not really change that in any way, though the bigger the setup, the more additional checks you'll probably want to add. You have X containers to monitor, but you had X server processes to monitor previously. If your use case made it ok to depend on just e.g. hitting a page on a website that depends on all X services being up, then your single old monitor still does the job.

The overhead can be made really minimal and beneficial even for a single server setup, yet help you if/when you suddenly want to scale part of it. Personally, I run about a dozen docker containers on my home server so far, and I expect that to increase several times over as e.g. every web app I experiment with ends up running in its own container, with the docker setup in a git repo as documentation of exactly which project caused me to pull in which additional packages etc., and keeping the host itself as pristine as possible. Makes me lot less nervous about distribution upgrades etc.

And how does running it all in the same container make it easier to discover? All services listening on localhost?

I don't think I understand your point.

From the Docker Host itself, if you need to manage the state of a container, the intuition is that you need to go into the container (with SSH) in order to do so. But by externalizing your state, you can manage it without the need to enter the container. Assuming your Docker Host is secure, this doesn't make anything less secure just because you're no longer abusing SSHd in order to manage your application's state.

In the case you need to gdb, or strace the process, you can do that from the Docker Host with nsenter. Assuming your Docker Host is secure, you no longer need to abuse SSHd to carry out a debugging task that has nothing to do with needing a secure shell.

Neither of these use-cases have anything to do with the security of SSH.

In the case that you need to do these things from a remote host, the prescribed answer is indeed SSHd to access the Docker Host, at which point you switch to the previously suggested methods for managing state.

"I don't see how granting access to the host is a cleaner architecture... from a security standpoint, it seems the opposite."

Because now you only have to worry about one security layer instead of N security layers for each container you run. The security layer is now actually coupled to the act of granting access to the host, its intended purpose vs granting access to a container so you can manage its state or debug it or whatever.

As far as being locked into Docker's APIs, I totally miss the aim of this remark. Volumes are just paths on the filesystem. If you're talking about the interoperability of standard tools to manage your state, I don't think they will have problems in this case.

> I don't see how granting access to the host is a cleaner architecture... from a security standpoint, it seems the opposite.

If you are concerned about granting access to the host, you should be concerned about granting access to a Docker container, especially as root, until it's far more battle tested.

And if you are concerned about this, there's a simple solution: Group your Docker containers in KVM vm's.

If you are not concerned about this, you can easily do as he suggested, and force "nsenter" on login. Or you can do both: Group your containers in a KVM vm and force ssh into the KVM to use nsenter into the appropriate containers.

Part of the point is that it makes little sense to add to the complexity and attack surface to add a process monitor per container (to run both sshd and the app) and an sshd per container, even if what you add is something that provides "industry standard, encrypted communications with proven cryptography" if it is not actually needed inside the containers.

You're also not getting "locked into" Dockers API much if at all. What is proposing depends on two things: A way of executing a command that can access a certain part of the containers filesystem, and a way of entering the appropriate namespaces.

Both are easily abstracted behind tiny (1-2 line) scripts where you can replace "docker run" and "nsenter" with ssh if your needs change. And one (nsenter) is entirely independent of Docker - it will work with e.g. lxc and anything else that makes use of cgroups as well.

I don't think it's SSHd in particular that he has a problem with. Docker maintainers do not seem to like the idea of people treating Docker containers as lightweight VMs. They seem to want images to be restricted to running as few processes as possible - ideally one. I think they view it as too hard to scale once you start going down the path of running multiple services in an image - I've seen the "pets vs cattle"[1] analogy used for explaining why.

There is an image by Phusion called baseimage-docker[2] which adds SSHd, init, syslog, and cron in an attempt to make Docker containers more like lightweight VMs. But in the #docker channel, I've seen people have issues with it. For example, one person had some /etc/init.d scripts that wouldn't start up (other ones started up fine). Turns out that one of the signals that init was waiting on to start that script was never getting sent (I think it was networking coming online?), and that was just a side effect of how Docker works that couldn't easily be fixed. The Docker maintainers in the channel discouraged using this image for these reasons.

[1]: https://groups.google.com/forum/#!msg/docker-user/pNaBYJkmnA...

[2]: http://phusion.github.io/baseimage-docker/

I run sshd in one of my Docker images. BUT most of my Docker images are single-app, with no sshd. This includes the various applications I'm working on. The one I ssh into is a (most of the time) replacement for ssh-ing into my actual server. It defines /home as a volume that I pull in from a directory on the server.

Then when I start playing with a new application, I set up a new docker image for it, and has that docker image bind to a suitable subdirectory that is then shared with the "dev" image I ssh into.

This means I always have 100% up to date documentation of the requirements for running the app, in the form of the docker file.

I don't see a problem with running sshd in a Docker container. I do have a bit of an issue with "throwing a bunch of stuff" in the containers. If your purpose for having ssh there is to be able to investigate things, then I'd argue you don't really need it in most cases:

- You can copy files out of the containers with "docker cp"

- All the processes are visible and traceable from the host

- You can choose to export the important bits of the filesystem of the container, so that you can start a second container to inspect those bits more thoroughly (this can also be used to great effect to allow apps that needs to work on the same file sets to be isolated in separate containers).

- During dev/testing you can easily override the entrypoint and start a shell.

In practice I've found that while introspection of various forms is important during testing and development, there's also a certain benefit in not giving yourself direct access to easily modify stuff: It makes it easier to maintain discipline and update the Dockerfile etc. and rebuild containers.

If you also use e.g. systemd to run the containers, and log to the console, you get your logs concentrated via journald and optionally shipped on to rsyslog etc., and combined with the docker tools and careful use of exported volumes, there's very little state left where there's much need to ssh in to a specific container. There's also a tool (forget the name; EDIT: D'oh, it's "nsenter" that's presented in the article I'm thinking about) that let you run a process in the same cgroup namespaces as your container as well.

If anything, I'm leaning towards trimming the containers further, down to the bare essentials for each specific app.

I don't think your base image misses the point of Docker. Different people use Docker for different purposes, that is normal and a fundamental goal of Docker.

I do have criticism for your communication around that base image, starting with the link-bait blog post "you're using Docker wrong". Your message is that anybody not using Docker your way (full-blown init process, sshd, embedded syslog) is doing it wrong. That is not only incorrect, it contradicts Docker's philosophy of allowing and supporting more than one usage pattern.

My other criticism is that you point out a known Docker bug (the pid1 issue) and use it as a selling point for your image, without concerning yourself with reporting the bug let alone contributing to a fix. Meanwhile many people hit the same pid1 bug and have reported, suggested possible fixes, or contributed code to help implement that fix. If you want to be taken seriously in the Docker community, my recommendation is that you consider doing the same.

This, to me, sounds like the much more reasonable and sound approach to containers, as opposed to the "SSH bad!" article. As someone who doesn't use containers or virtualization heavily, but maintains multiple systems (servers, desktops, phones, etc), I can unequivocally say that SSH is practically a hard requirement. And just having it as the way to get to a container seems like a no-brainer, in that you don't have to learn yet another tool, that might not work, and even if it does, won't cover all the use cases of SSH. I use SSH for remote admin and automated backups of everything (phones, tablets, servers, desktops, etc). Adding another thing to be backed up via SSH is easy, no matter if it's virtual or not.

EDIT: I do really like the separation of concerns and modularity that are brought about by the approach advanced in the article. But I would argue that the arguments against SSH apply many times more strongly against alternatives to it: security upgrades? You're going to have to do that much more often with whatever you use to replace SSH. SSH has proven track record for security and authentication, it's well known, lightweight, and generally doesn't break on its own.

Thanks for your efforts with creating/maintaining baseimage-docker. It saves me effort from needing to maintain my own baseimage, and has given me some interesting ideas to try for my docker images.

I've now introduced and put docker-based infrastructure projects into production environments at 2 different companies, and IMO having sshd in the containers has made it much easier and familiar for techops/devops teams to get started with docker.

Docker-attach is a much more limited solution, and I think that introducing another tool like nsenter is a non-starter since it just adds more complexity with additional tooling and dependencies. Another tool when ssh works? The additional cpu/ram use isn't a big deal, and for security as long as I secure sshd and my keys/password properly (not storing them in my image, for example...), no worries.

Docker logging is also limited compared to tried and true linux logging utils.

Docker process supervision is still a bit immature and unreliable. I'll keep trying the built-in solutions, but I have everything working fine now without needing to wait for subsequent Docker releases.

Docker is a really convenient wrapper around a bunch of standard Linux tools, and IMO that has been its power. The weaknesses in Docker have been where it tries to build its own replacement for existing and mature solutions (logging, supervision, networking, etc.).

A lot of the functionality of libcontainer, libchan, libswarm seem to be done by existing tools. Why reinvent the wheel? Are the existing project maintainers unwilling to take pull requests?

> Microservices Are Not A Free Lunch

I don't buy that article at all. There's a lot of strawmen there where things are suddenly needed for micro-services while they apparently aren't when running the exact same things inside a single VM.

You can architect a microservices based system in ways that add operational complexity, but if you do it should be because there are substantial benefits to be had that way.

But you can equally well take that monolithic VM that seems to be what that article is assuming the alternative is, run Docker inside it, and run each service in a Docker container inside it, and still start to realise substantial benefits; not least because it makes it easier to grow out of the single VM easily as/when needed by making dependencies much more explicit and allowing each services software dependencies to evolve separately.

I agree you don't have to use Docker only that way, but the more I've played with Docker the finer-grained I end up making things...

Couldn't agree more with this sentiment. I've started using docker to replace vagrant/VMs for my test workflow. I use it to spin up a container, provision our server code via chef, and test out various changes.

Is chef my favorite too? No. Could I be using docker in a more optimal manner? Sure. But the reality is that I wanted the simplest possible path to integrate docker within our workflow, and it already saves a ton of resources. I run linux on a machine with 4G of ram, so believe me, utilizing containers for testing infrastructure changes is a huge improvement.

So docker as a lightweight VM is most definitely a valid use case, IMHO

Would you not mount nfs volume in each container to share data?

nsenter is part of util-linux, which means it will eventually make it into pretty much every Linux distro around. The functionality doesn't belong in Docker, because it isn't Docker specific - all it depends on is cgroups, and cgroups is a kernel feature, not a Docker feature.

Once it's in there, a "jexec" equivalent would a couple of lines of shell script.

You need not just accept that the ssh daemon itself is safe, but that:

- Your key management is safe. - The process manager you now need to introduce to start sshd and the app running is safe. - That the ssh daemon is sufficiently protected against abuse. - That your configuration of it is safe.

If you don't need ssh in every container to do achieve what you need to achieve, why do you want to have to deal with each of those and waste the extra resources of having a bunch of extra sshd's and process monitors running?

(To the last point: Yesterday we suffered an attempt at brute-forcing ssh on a public facing server. We're used to people trying to brute force passwords. But as it happens, it is "easy" to make openssh consume all of your servers resources if you don't block access on the network level in the event of an apparent attack; so if any of those ssh servers are reachable in any way from the outside, you have just increased your attack surface even if your key management and everything else is perfect and they have no way of actually getting in)

I recommend you read the submission, then!