EKS Security Checklist: 10 Best Practices for a Secure Cluster
cast.ai
cast.ai
* Accessible to someone inside the pod
* Not Accessible to other pods in the cluster without cluster admin
* Can be retrieved outside EKS from API/ETCD or AWS Secret with enough permission
* Persistent across the pod lifetime (you cannot delete a secret file after use: https://stackoverflow.com/a/54404149/1997873)
Is the only reason left for preferring files is that integrations prefer it as they can easily implement it as init pod? or that they have less chance to appear in ENV dumps in logs?
My take is that it is noticeably more secure to use the cloud provider secret manager (or hachicorp vault).
There’s not much difference between a configmap and a k8s secret. The secret doesn’t have a separate decrypt operation so you have to lock down even list access and can’t separately view it’s metadata so you don’t get much benefit from k8s anyways.
If you use a k8s secret adapter to the external secret manager tool rather than loading directly in the container out of band, the security picture gets more complicated.
How would you do that? They still are in /proc/, aren’t they?
Even though Secrets and ConfigMaps are used exactly the same way, there's a big difference between them. The most obvious one being that Secrets contain values that are sensitive. Of course, by the time these sensitive values have made their way into a Kubernetes Secret stored in a Namespace, they appear in cleartext. But the point of a Kubernetes Secret is to offer cluster administrators an Object that is explicitly meant to store Secret values, which can then, _for example_, be stored in ETCD differently (encrypted at rest for example).
> My take is that it is noticeably more secure to use the cloud provider secret manager (or hachicorp vault).
Yes, you should use Vault (or whichever is your favorite secret management tool), but this has no bearing on the Kubernetes side: ultimately, your sensitive values will end up inside a k8s Secret object, that is then being used by your Workloads (i.e. Pods).
> If you use a k8s secret adapter to the external secret manager tool rather than loading directly in the container out of band, the security picture gets more complicated.
Loading sensitive values out of band should be avoided since it is non-standard and forces your users to implement a method of retrieving secrets (which will be 1- painful, 2- buggy, 3- a security incident soon).
From the point of view of the process running inside the container, there is absolutely no difference between a sensitive configuration value and a non-sensitive configuration value. When this value reaches your process it is in cleartext.
What matters is the transport and persistent storage of your value. For a non-sensitive value, the storage can be done in VCS and transport over non-secure channels. For a sensitive value, the storage must be done encrypted (in hashicorp Vault for example) and transport be done over a secure channel (with authentication/authorization).
By using an adapter ([0]External Secrets for example), you actually abstract away the transport and storage of secret values from your users; allowing them to treat all configuration values the same (by reading an ENV variable, or a file), regardless of whether they are sensitive or not, and regardless of where they are stored at rest.
It is possible to not use K8s and make things just as convenient. For example by encrypting the secrets and putting them into the docker image and having a tool run at startup that decrypts them into the same file format as K8s (or as env variables). More equivalent and with less exposure of encrypted values would probably be to run a sidecar container that pulls in the secrets from the secret manager. If you want to represent the secrets in K8s you still can as a CRD that doesn't actually expose the secrets when listed or read.
What is the issue with Kubernetes Secrets that makes you want to go through all this complexity and avoid using a facility explicitly meant to manage secrets efficiently?
> _for example_, be stored in ETCD differently (encrypted at rest for example).
Cloud provider k8s clusters are deployed with encryption at rest for ETCD now. If secrets were doubly encrypted at rest that wouldn't fix how their secret values are exposed in what should be just a metadata listing.
> From the point of view of the process running inside the container, there is absolutely no difference between a sensitive configuration value and a non-sensitive configuration value. When this value reaches your process it is in cleartext.
This model is desirable for convenience, but not for security. In order to securley handle a secret properly, the process needs to know that it is handling a secret. This can help to ensure secrets are not accidentally logged. For higher levels of security, it also helps to periodically refresh secrets so that they can be given as expiring tokens.
Files on the other hand has access control as a primary feature. They carry meta data that makes it obvious which uid can access the data and which gets included with backups and storage.
Apart from that, there are also possible encoding issues of secrets in environments. Secrets must be kept verbatim for integrity reasons better more suited for binary formats.
Then there are the size restrictions. An environment of several kilo- or even megabytes seems plenty until for some ridiculous reason it's suddenly not, with not at all obvious results.
Another approach that sometime is supported, is to use a proxy that handles secret based connections. Like a MySQL pod proxy that you connect to which take care of reconnecting to his upstream on secret change.
Both are transparent to the app pod (and its developers)
What's the best course of action if you need to connect to a specific pod? (You might want to do this if you're connecting users or to the same server instance, for example). Managing reverse proxies can be pretty cumbersome if you want to autoscale. Exposing an external node IP is easiest solution but if its a security risk, what are the alternatives?
A good article about it is https://aws.amazon.com/blogs/aws/new-port-forwarding-using-a.... Since that article they introduced the capability for forwarding to the remote host by using the document written above and by adding the host key and value in the parameters.