The Docker security philosophy is “secure by default”
blog.docker.com
blog.docker.com
I don't want to criticise Docker here from a security perspective - several security features in rkt are based on the implementation in Docker, and running your apps isolated under Docker is definitely more secure than running them all on the same unconfined host.
(Disclaimer: I work at CoreOS, though not primarily on rkt)
One of the big challenges with the containerization landscape is keeping up with changing product capabilities as there's a lot of movement all over the place.
However, where is image signing? Where is pulling images by their hash instead of mutable tags? Where is restricting what images run according to their hashes? Where is Docker not running a daemon as root? How about Docker doing more hardening releases and fewer feature releases?
There's a lot of low hanging, security related fruit which haven't been picked. Why not?
Process isolation is a fantastic. There just really needs to be more, in my opinion.
The coolest bit here is to be able to do threshold signing. Essentially k-of-n signing for containers and verification gates that only allow containers with enough signatures in order to deploy. For some more background check out the blogpost here: https://blog.docker.com/2016/08/securing-enterprise-software...
Disclaimer: I manage security at Docker.
> "Secure by default" is a great philosophy to have. I hope Docker continues to push into this direction.
Thank you. We will absolutely keep pushing.
> However, where is image signing?
Docker supports image signing out-of-the-box. It's based on TUF which is the state if the art in secure content distribution. Along the way we open-sourced our underlying TUF implementation so thaf others can reuse it: https://github.com/docker/notary
> Where is pulling images by their hash instead of mutable tags?
That's also available out of the box: "docker pull NAME@DIGEST"
> Where is restricting what images run according to their hashes?
That's not available out of the box, but a popular way to implement it is to funnel images through a trusted private registry, only allow pulls from that registry, and then controlling what you push to it.
Another possibility is to use the new auth plugin system, although I haven't tried.
> Where is Docker not running a daemon as root?
Docker runs unprivileged by default, but only on relatively recent kernels with user namespace support (it drops itself to "fake root" so it can still create containers without actually having uid0).
On older systems (or if you don't trust user namespaces) I agree we could break up some parts of the daemon (for example push/pull) to drop privileges further. That requires serious refactoring which we have been working on gradually for some time.
> How about Docker doing more hardening releases and fewer feature releases?
We follow the linux methodology: every release includes many hardening improvements, as well as whatever features have been deemed stable enough to merge. We don't control what pull requests the community sends, but we work hard to maintain a high quality bar and encourage quality-oriented contributions. The Docker core engineering team invests more time on hardening than features.
> There's a lot of low hanging, security related fruit which haven't been picked. Why not?
I'm sure there are :) Security is a process, we will never be done. The important thing for us is to take it seriously, invest appropriately (Docker core team employs 6 full time security engineers and funds various third-party security efforts), and make sure we are perpetually improving.
On top of that, Docker is known for making powerful technology more accessible to more people. We want to use that "superpower" to make security more usable by non-experts. We think that contributes to making everyone more secure.
> Process isolation is a fantastic. There just really needs to be more, in my opinion.
Docker security is about much, much more than process isolation.
Thanks for the feedback. We hope to see you on the github repo, we value bug reports :)
Sounds great in theory, but there hasn't been much traction on that project, at least from their own website, for over 6 months. Not much in terms of peer review of their method, or practical audits of their technology. Even your own audit is over a year old now, despite constant changes to the codebase.
That said, I do recall now when this was announced originally, and I'm glad that it at least exists. That it has skipped my memory makes me wonder why there has been little to no press about it since then. How broadly adopted has this been? Can we get signed images for, say, Ubuntu from docker hub?
> docker pull NAME@DIGEST
A great feature to have. Can you point me at the digest for the ubuntu repo on docker hub?
> serious refactoring which we have been working on
Good to hear. Also, interesting to note that runc still requires root access to make cgroups; I was not aware of this restriction.
> The Docker core engineering team invests more time on hardening than features.
I find this to be an interesting position to take, considering that a core component of docker has been historically re-written with every other release: registry in 1.6, disk plugins in 1.7, networking in 1.9, runc in 1.11. IIRC, networking was re-written again in 1.12, to support yet another version of clustered networking. That's a lot of churn for less than a year and a half worth of time.
> We hope to see you on the github repo, we value bug reports
That hasn't gone well for me in the past.
devicemapper errors in ubuntu 14.04, prior to dynamic linking: not our problem.
devicemapper dropping mounts sporadically: move to overlayfs.
Low level mutexes being shared between containers: that's nodejs' problem.
It's frankly become easier to create and use workarounds like spam-unmounting orphaned devicemapper mounts, or custom-compiling and distributing docker, or disabling docker-managed networking, and so on.
I feel bad for ragging on Docker like this, especially since I do end up using it frequently; but as a system administrator Docker is a nearly daily headache. Our developers love it, but the rest of us... not so much. I guess it's one of those "technologies everyone complains about" vs "technologies nobody uses" things. It's just really frustrating.
export DOCKER_CONTENT_TRUST=1
docker pull ubuntuHowever, there's some usability problems here that I'd like to bring to your attention:
- There is no indication that the pull is different; no output from trust verification that it is indeed signed. This means I have to trust that Docker did the right thing, with no means of verification.
- Inspecting the image after pulling gives no indication that the image is signed, and gives me no way to do my own signature validation on the image.
- It does fail properly when pulling an unsigned image. Yay!
- Docker run initiates a connection back to notary, and fails if it can not connect. This makes me uneasy - it makes me wonder what data is passed, how it's being used, what changes are being made according to the response, and so forth.
- Using trust is a per-command decision, instead of a daemon setting.
- There's no clear method to revoke a signing key if identified to be malicious.
- This seems like a good candidate for "secure by default" once some of the usability issues are resolved.
To be fair, runC was its own project for quite some time. Docker just switched from importing the runC code to actually running the binaries through containerd as a proxy in 1.11 (a design decision I'm still trying to wrap my brain around, but that's a separate issue).
> Docker runs unprivileged by default, but only on relatively recent kernels with user namespace support (it drops itself to "fake root" so it can still create containers without actually having uid0).
Is that right? As far as I'm aware, runC doesn't have support for anything like that (and Linux's method of handling network namespaces under an unprivileged user namespace doesn't allow you to even have a network connection to the outside world). Sure, in runC we're working on rootless containers, but that's a while away from being done and Docker might not ever be able to support rootless containers completely (since it needs to handle networking and some other things that still need some quite significant kernel work to complete).
https://github.com/docker/docker/issues/13490
It's not 'secure by default' if secrets are flying around through the system with no formal way to prevent it.
If you're playing to the 'secure by default' crowd, they aren't going to just trust that you're on the case if you don't communicate. Platitudes are something people watch out for when security is on the line, and they'll chose another solution if they think you're full of shit.
You have to show, at all times and all things, how you're thinking about the problem. 15 months of dead air is a lot to atone for.
Gartner is like a financial ratings agency. I wouldn't rely on their recommendation if I were concerned about the security of any application or platform.
the other thing to note is that when deploying Docker engine the defaults tend to go towards making everything work and aren't necessarily the most secure, so its very much worth doing some additional hardening where possible.
Specfically things like
- enabling user namespacing
- disabling icc
- removing capabilities (e.g. NET_RAW)
- not using the default docker0 network
- disabling userland proxy
There is a CIS standard which is pretty much up to date and has some, hopefullly, decent sets of things to look at hardening (disclaimer I helped work a bit on the latest version...)
edit: found the link - https://docs.docker.com/engine/reference/commandline/dockerd...
Looks like Docker have put out a content scanner service to mitigate the problem this still seems to be a weakness relative to the pre-Docker way of building VMs.
Actually for all people below, here are my reasons:
Docker adds another Layer. This also adds another attack target (docker). Saying a software is more secure while adding another layer is just naiv.
Docker is good for some companies and it definitiv is true that you can make it to run a PaaS, it's still not more secure as the world before.
Also some other people also pointed out other things. With docker you add a lot of complexity to your actual software, that is negligible if your big since you have a total different set of problems.
Still when it comes to docker it's mostly buzzword bingo. I mean just looking at the table... oh god it's like half the things there could just be done without docker in a way more secure way, without LXC, Docker or RKT.
Also I always tought that docker should make the deployment easier, which would be a hugh bonus in security since the process of understanding and everything beyond would be more understandable. however the more docker grows the worse the complexity gets. Docker once looked shiny but than the enterprise hit it and it's direction more and more makes it less secure since they try to add more and more stuff and stops focusing on the stuff docker was firstly invented.
How could adding a ton of abstraction (which grows every release) making something more secure? I just keep that as an open question to everybody.
I don't mean to sound adversarial - but what's your counterpoint to that?
Has there been significant improvement in this area lately?
I'm not sure I agree, I'd offer some improvements:
- Gartner "studies" are usually paid advertisements
- The comparison table is missing columns for DOM0/1 VM's and BSD jails
- Encryption in-transit or at-rest isn't mentioned
- Nothing about key management or key rotation
- Non-mutable / exported log files aren't in the defaults
Docker 1.12 in swarm mode, for example, does automatic key rotation and issuance of the TLS certs assigned to every node in the cluster. These certs are used for automatic TLS between every node in the control plane of the Docker Swarm. This is all automatic and transparent to the user -- no manual management of certificates is required.
Now that we have cryptographic identities assigned to every node in the cluster we can use that to build secrets/key management in to the system.
Additionally Docker 1.12 is swarm mode has overlay networking with encryption possible for container to container communications.
In terms of logs, Docker supports log drivers that allow all logs within Docker containers to be exported to off-host logging services.
You have to upgrade all containers, which requires figuring out which ones need upgrading.
The concern isn't generally that someone will escalate and then delete some other container's filesystem.. it's that once they get remote execution in a container that has connections to your database, http to your discovery service etc, they can steal your customer information.
I will say however that this post is showing that by using docker or frankly other well manicured execution enviroments, large swaths of attacks will hopefully not work (selinux & seccomp being great examples).
The other benefit of short-lived containers is that any given vulnerability only lasts as long as the container (assuming you're rebuilding your image with the latest & greatest software/libs before each deployment). Whereas with a full OS/VM you have to find down time to patch that sucker and that might take a month or even a quarter or longer.
Even if you keep deploying new versions of containers every few days because you use them to deploy new features, how about the db and web server containers? And how about the vulnerabilities that won't be patched until you make a new deploy? We should build a new container after each apt-get for security patches, right?
On the other hand, if you architect your database usage in such a way as to ensure that cutoff transactions get detected and retried without relying on persistent connections it can be a good idea to use containers for hosting your database(s). Of course, your use case needs to fit this kind of model for it to work. If your use case requires mostly serial transactions and can't work with "eventual consistency" then containers are probably a no-go.
Yes. That's the idea. Here's when and why you re-deploy:
* A new, updated version of software/a package is out.
* You've made changes to your app. Even something as simple as a typo.
* Your container has been up too long (e.g. 24 hours).
The whole point of containers is that they can be brought up and down in an instant without impact (if you architect things correctly). So even a VM that runs "apt-get update ; apt-get -y upgrade" every night may not be as up-to-date or secure as the same software running inside a container.
Not sure how much this affects what's in the wild, though - I mean, I have no data to show that people really do tend to upgrade more often while containerized than not.
It is important to have an inventory so you can do patching across the board and have notifications that it's needed. This is why there are now a number of security scanners for Docker containers, including Docker's own: https://blog.docker.com/2016/05/docker-security-scanning/
Disclaimer: I manage security at Docker.
Personally, as a sysadmin, I dislike docker. Not that there's anything wrong with its implementation in particular, but because I can no longer confidently say what is actually in our stack (we run 4 different versions of Python, last I checked -- it's absolute madness).
Now, I grant Docker's argument: if you're going to do something like that, Docker is pretty much the best way around to do it. Fair enough: it's the best available tourniquet for the self-inflicted wound of bad stack management. But to me, the ease of redeploying gets outweighed by the increased complexity of the security reasoning you have to do.
Docker containers run with default seccomp profiles, namespacing (filesystem, PIDs, mounts, etc), LSM policies (AppArmor, SELinux), and capability dropping.
These are all common sense security controls that aren't widely available because they're typically hard to use. Docker makes them defaults for everyone. This makes classes of remote attacks against applications far more difficult. Tangible example: read-only filesystems + mount namespacing make vulns that require filesystem modification or directory traversal far more difficult.
At this point arguing against running in Docker is like arguing in favor of IE6's security model instead of Chrome's.
Disclaimer: I manage security at Docker.
* --read-only
* --security-opt="no-new-privileges"
* --cap-drop=ALL
* --net="none"
* --cpu-period=
* --cpu-quota=There is also --pids-limit=<some number> against fork bombs.
EDIT - this git repo has more links to security related articles etc. https://github.com/wsargent/docker-cheat-sheet#security
The reasoning that "more layers of abstraction necessarily equals less security" does not seem true to me. Granted, it increases the "surface area" that is exposed to attacks, and there's indeed a tradeoff involved, but this tradeoff may swing either way - either it does pay off or not.
Taking this reasoning to the extreme, there are plenty of abstractions that pay off for secutiry. One simple example is that software written in Java (for example) is by default more buffer-overflow-resistant than software written in C, due to managed memory/bounds-checking. In this case, the tradeoff was performance vs security.
> How could adding a ton of abstraction (which grows every release) making something more secure?
One example is the "buffer overflow resistence" in Java vs C mentioned above.
Actually trust me it's way easier to control a single stack instead of managing a stack on top of another. Especially networking is no fun. Maybe it's cool and easy / shiny if you can have everything in the cloud, but that just won't apply to everybody.
There will be different classes of bugs, of course, but I find it hard to believe that it will be more bugs, or more serious bugs, or most costly-to-fix bugs. Otherwise, your own company would have decided to use C instead of Java, right?
> Hell you can start your programs under Linux with user namespaces just without docker
Sure you can, but if Docker (or any other solution, for that matter) makes it easier or faster to do that, more people would do that. There's a more or less fixed number of work units to dedicate on a certain project, if you have to use them all to set up security solutions, your lunch will be eaten by other people.
> most stuff is just possible without docker and the big abstraction
Most stuff is "just possible" without any abstraction at all. However, one has to learn to leverage abstracted-away stuff, otherwise one will spend a lot of time and accomplish very little valuable stuff.
> trust me it's way easier to control a single stack instead of managing a stack on top of another
Easier does not equal safer, faster, or more business-valuable.
> it's cool and easy / shiny if you can have everything in the cloud
Not sure where "the cloud" enters the equation? You can use Docker (or whatever else) to manage even your on-premises stack.
---
I don't know the specifics of your situation at all, but if your only argument against a particular piece of tech is "it abstracts away stuff, therefore it is worse", you might want to consider some self-reflection on whether you're not just suffering of NIH Syndrome (https://en.wikipedia.org/wiki/Not_invented_here)