Introducing Docker Secrets Management
blog.docker.com
blog.docker.com
Building it a second time was interesting. One of the biggest reasons why Keywhiz didn't go anywhere was the fact that it is incredibly hard to setup, and requires you to bring your own PKI. This time we didn't make that mistake and integrated directly into Swarm, which is the right place for it to live, and turns setting up your own PKI into a one-liner.
Anyway, AMA.
Disclosure: I work on the Docker Security team
With this you can securely store the certificates; takes away the developer's responsibility of taking care of managing and distributing them to their apps, swarm will take care of that in a secure end-to-end fashion.
Docker-containers have many advantages, but one of the big drawbacks until now was that it was really hard to pass secrets into them. Imagine you are a simple, stupid Java-container. You handle HTTP-Request and need to access other services like Databases in order to function. In order to connect to the postgres-instance you need to know 3 things: the URL, the database-name and a password. Getting the URL and the database name is easy, just pass them via environment-variables. The tricky thing is how to pass the secret. Environment variables are not encrypted and super easy to read as soon as you have gained access to the container. There were some solutions, but they were complicated. So most just passed them via env-variables. Huge security risk. Very bad.
This solves the problem.
Or think about you exec'ing imagemagik and now the process running potentially adversarial code also has access to your parent's env.
Or think about an application crashing and doing an unintentional core dump to disk.
If it's static, follow up to that would be, how are changes propagated to already running instances?
Is there a way to restrict access to reading said secrets once the container is running? Say you're running possibly malicious code that has access to the local filesystem (ex: CI test runner) is there a way to restrict a process from reading those files? Can we simply delete them (i.e. "burn after reading") or are they fully virtual?
I don't know anything about docker, but the best way I found to do this in linux in general was aa_changehat() [0]. You write an apparmor profile for startup and a sub-profile for the running app/service. After setup, you call aa_changehat() to switch the current process to use the subprofile. You then throw away your magic token, so there is no way to switch back.
You don't even have to link to libapparmor, under the hood aa_changehat() just writes some string somewhere in /proc, so you can replicate that. Note, I haven't actually done this, but working on it right now.
[0] http://manpages.ubuntu.com/manpages/wily/man2/aa_change_hat....
Can I plug in my own secret store?
It is the most important step that you can package containers without having to know the production secrets and to have a "standard" was to retrieve them.
No one, without production access, has a way to obtain them.
> Currently, anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet
> If multiple replicas of etcd are run, then the secrets will be shared between them. By default, etcd does not secure peer-to-peer communication with SSL/TLS, though this can be configured.
We usually recommend subdividing the node acls by namespace when running disjoint node sets (where tenant A can't schedule onto tenant B's nodes). More fiddly than it has to be in Kubernetes today.
The real solution is of course to limit what nodes can see - I won't say it's trivial, but it's an O(1) check based on the pods scheduled on that node.
That's partially why we (openshift) just don't allow containers to run as root by default - it sucks for new users (most images assume root) but it dramatically lowers the risk of compromise across the board in any scenario with multi-tenancy. Agree that with isolated nodes per tenant you wouldn't have this issue, which is a more straightforward RBAC story.
- If you have access to the managers, you have access to all the secrets (and access to administer the whole cluster), but normal swarm nodes only have access to the secrets required to run services that will be scheduled on them. Also this are erased as soon as the service is deleted or rescheduled on some other node.
So far my approach was defining environment variables in the various docker-compoose files (in a separate deployment git repo), but this looks like a really nice alternative.
Do you have plans to update the library images to give us a choice between using ENV and secrets (for DB server passwords and the like)?
On an aside: I've gotten the Docker Datacenter announcement mail today. I only took the time to skim its contents quickly and at first thought this was a DDC-only thing. Glad to hear it isn't, keep up the awesome work.
edit: clarified my docker-compose usage
- Exposing secrets as in-memory files has a lot of advantages over ENV variables (harder to leak).
- We already started updating a few images (MySQL, for example), so they can use Docker secrets.
- Definitely not DDC only, but note that RBAC over secrets is a feature of the commercial product.
You have to argue that Docker uses the Linux isolation mechanisms that make those containers virtual machines in the sense and spirit of PCI.
Treating containers as VMs makes some other requirements even easier like the request to have a minimal system and to only have one function per server - thats how you want containers to work anyway.
(Btw PCI has nothing to do with PII.)
I'm curious if you can relate what his reasoning was?
Docker is, among other things, a wrapper around a bunch of Linux kernel functions, the likes of which have been used for many, many years by companies like Google to facilitate all kinds of useful isolation.
Use it for for operational isolation, not for critical security isolation.
At the same time, going back to the initial quote:
> solution architects from Amazon and he wasn't sure if Docker is ready for PII data
A lot of companies are using containers to execute code that manages all kinds of regulated data right now, ya?
The security limitations of Linux namespaces and friends are mostly related to the execution of untrusted code.
Consequently, because it's mounted as a filesytem, what if the service is compromised and vulnerable to arbitrary code execution, directory traversal, etc? The secret could then be leaked.
Am I misinterpreting something? How would others here handle this?
Edit: To clarify: rotating a secret will cause the service to restart. So I guess by "doesn't seem ideal", I mean it doesn't seem like an option.
If a service is compromised, you should always assume the secret is compromised.
I like this idea. As a benefit, you wouldn't have to rely on developers of individual services to make sure the secret is cleared. This would also have the benefit of causing an error if something else happened to read the secret before the intended service, indicating a possible compromise (or maybe just a misconfiguration).
(I use a similar concept with `xclip -l' to allow reading passwords from the clipboard a single time before it quits.)
We have been testing out production clusters with Swarm 1.13 and I think its neat. But what do I know - I'm not a cloud hosting specialist. And the lack of success stories out there is making us nervous.
Depending on the size of your team, I would recommend Kontena (www.kontena.io) or Kubernetes (https://kubernetes.io/). While I haven't used the latter, a group of two devs is re-architecting our entire system to use Docker and Kontena. Nearing the end, the whole process seemed incredibly fast to iterate. While K8s seems far more proven in real production environments, it seemed much more daunting to us with more features than we needed for our use case.
secrets stored in plain text and (by default) transmitted in plain text between etcd services, does not fill me with confidence on their security.
If setting up a secure cluster is daunting, then use a distribution that handles it for you. OpenShift (https://www.openshift.org/) is built on kubernetes, and it's install is secure by default.
Disclaimer: I work for Red Hat, and spend lots of time on OpenShift consulting.
It's definitely something we'll fix in Kubernetes, but rooting workloads is the primary problem, and secondary acl defense in depth is good but won't block most attacks for long.
Default ACLs are clearly the most important line of defense in an orchestrator's security model, because whether a container escape can happen is not something the orchestration system has control over.
Being able to trigger node compromise should have nothing to do with being able to schedule.
there are tons of these niggling issues that are cropping up.
the complexity of using k8s goes up exponentially every day. I have bo doubt it is a great piece of tech.. but at this point, it seems tailor made for consulting.
On OpenShift I'd be very interested to see a list of the secure defaults that have been chosen, is there a list of the changes from base Docker/Kubernetes policies available anywhere?
Anyways, my issue is: Does Swarm integrate a solution for URL routing and if so, does it tackle the persistency problem?
https://success.docker.com/Datacenter/Apply/Docker_Reference...
Hope this helps!
I'm a little worried about two aspects of what has been shown. From the article it shows:
$ docker exec $(docker ps --filter name=redis -q) ls -l /run/secrets
total 4
-r--r--r-- 1 root root 17 Dec 13 22:48 my_secret_data
I'm also a bit worried that the default protection on the file has it set to world-readable. Since it appears that secret distribution is independent of the container setup itself, there doesn't appear to be any way of setting ownership and permissions on this file. That is, if one were able to chmod/chown the file in the Dockerfile, running a `docker service update --secret-rm` and `docker service update --secret-add` would reset such 'fixes'.
A great start, and I can't wait to start using it.
$ echo "my secret" | dd bs=512 conv=sync | docker secret create my_secret_data -I'd much prefer passing a secret as an environment variable if that can be done securely. It's possible with some tools, but not out of the box with Docker itself.
eg, with one tool, you can do:
# docker run --rm -e PASSWORD={supersecret_password} someimage:latest program
Then '{supersecret_password}' gets replaced in the container at runtime with the value stored in the tool or from an integration with a separate dedicated secret management tool like HashiCorp Vault, and value gets masked external to the container such as when running 'docker inspect' command.
The benefit is that you don't need to modify or maintain a lot of pre-packaged applications that read environment variables instead of looking to the contents of a file on the disk, so it just works out the box.
However, under ordinary circumstances, you may not want to pass secrets as environment variables in docker (or at least be careful about it). A 'docker inspect' command can show any docker user the environment variable and its value, if you don't have a tool to encrypt the contents.
You can use Vault(https://github.com/Boostport/kubernetes-vault) or extend Kubernetes easily for your specific secret as a replacement if you need extra security.
Then you need a mutual authenticated connection from the api server to etcd + encrypted etcd hard drives + networking policies that only the api-servers are able to communicate with etcd and your secrets are pretty safely stored.
This seems like a fairly difficult problem to solve with meaningful security confidence. The only easy way I can see this working is if you pre-define groupings of kubelets and specify which pods may run on which kubelets, and enforce secret access the same way. Without this kind of hard separation, any kubelet is a candidate for running any pod and needing any pod's secrets at a moment's notice.
Furthermore, this sort of separation would require some kind of PKI; you could not just trust any given kubelet to accurately claim which groups a member of. You'd need to provision the kubelet groups with a secret that proves their membership in the group, and we're back to the secret distribution problem again.
And while it's true that a given kubelet may not be running a certain pod at the moment (and thus doesn't need the secret), that does not seem to be a hard security boundary. An attacker who controls the kubelet can manipulate which pods run on it, such as by terminating pods until the desired one is scheduled on it, or falsely advertising a huge amount of available resources to entice the scheduler to run pods on it, and so on. Ultimately if a kubelet is a candidate for running a pod, then it is simply a matter of coincidence whether it possesses the pod's secrets at a given moment or not.
That said, limiting kublets to have access only to secrets required by active pods will make things harder for an attacker, and so will provide value. But we should also evaluate the priority of that work in context of how hard it will be for an attacker to defeat that same restriction (e.g., advertise 1 petabyte of free RAM and disk, and 1000 idle CPU cores).
Coincidentally, I'm working on a project that uses Kubernetes and it has a very locked down pod placement policy, so the attack you described would be significantly scoped down. But I don't think the same is true of most Kubernetes deployments.
Securing nodes, preventing container escape, subdividing role access, constraining placement, limiting master surface area, and end to end audits have been the initial focus. Until those are in place, node secret access was less critical.
It is something that several folks are interested in working on soon.
Everything takes time to design and implement, and human power to work on the hard/not-fun things is pretty hard to come by in such hugely active projects...
Looking at your profile/comments you seem to be openly hostile towards Kubernetes. On that note: this is quite representative of the mentality I encountered while working there. Open Source is hard and bashing other projects is not something I consider to be fair game (regardless of whether the other side is adopting that stance or not).
Meanwhile, I admire Hashicorp's attitude of focusing primarily on improving their product without taking part in this silly "orchestration war".
(and thanks to cpuguy83 for having a more sensible view of what it's like to work on Open Source projects)
envdir /run/secrets <stuff>The article mentions a single master key for cluster encryption. Are there any plans to split this as in Shamir's Secret Sharing?
How do secrets and secret authorizations renew and expire?
Any plans for limited-use tokens/secrets?
tl;dr: author looks at secret management services and reviews them
KeyWhiz: Provides everything you need but with complex PKI management, meaning setup and maintenance is a pain. Secure.
Vault: A+ would test again. Awesome rotation policies, on-demand secret generation via backends, master key sharing. Legit and secure, everything you need but has to be configured on top of your cluster.
Docker: Super easy to use, and it's built in. 10/10 would use again. Keys encrypted at rest, keys encrypted over the wire, and shared with only nodes who need them. Secure all round
Kube: Totally insecure. Plaintext at rest, plaintext over the network, shared everywhere. Basically a plaintext POC
They currently debate over pluggable secret stores: https://github.com/kubernetes/kubernetes/issues/10439
You can use Hashicorp Vault as a third party resource to store your secrets: https://github.com/Boostport/kubernetes-vault
Kubernetes also supports RBAC to control access to secrets: https://kubernetes.io/docs/admin/authorization/
Quoting the docs at https://kubernetes.io/docs/user-guide/secrets/#security-prop...:
"Currently, anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet. It is a planned feature to only send secrets to nodes that actually require them, to restrict the impact of a root exploit on a single node."
As your cluster grows, your risk grows.
--secret="my_secret:MY_SECRET"
MY_SECRET=$(cat /run/secrets/my_secret) ./whatever