Secret Management in Kubernetes
redhat.com
redhat.com
In Kubernetes, secrets are base64 encoded, so they are quite secure.
The only protection I can think of is:
* The occasional glance at your screen.
* Injection attack (to some degree)
As Joe notes, this was just how you most easily encode binary data in JSON. We did discuss that the encoding has a minor benefit to “over the shoulder reader” obscurity, but that wasn’t the goal.
And if it's not obvious: Red Hat wrote enough of Kubernetes to know better than to base64 anything of value.
Some options I like:
* vault. Very flexible, you can have your app talk directly to vault and skip writing your secrets to disk. Or use plugins to populate k8s secrets from vault for consumers that can’t talk directly to vault. It’s a bit of a pain to operate vault and best practice is to put it in its own VM/cluster as k8s is not sufficiently isolated between processes.
* sealed-secrets / kops. You can encrypt your secrets with public/private keys and commit the encrypted blob along side your code. The private key lives in your cluster or deploy env. Has the strong advantage that secret-changes can be validated and deployed like any other code change, instead of requiring an OOB deploy process.
* your cloud provider’s KMS. Does what vault would do here, no operational maintenance is a plus. But you don’t get many of the fancy features from vault if you take this path. (GCP’s Secret Manager gives you a nice UI for managing secret versions though, for example).
I’m sure there are many other approaches too, these are just ones I’ve used at different times.
"Secret management" (i.e. having a non-ClickOps way to deploy secrets and a layer of security so that they're more than just not-configmaps) has been a black hole that has eluded our team for some time.
Should I use SOPS? Sealed Secrets? Vault? KMS? How does this integrate with our GitOps engine? Kustomize has no sensible way to pass secrets built in. ArgoCD actually has to be rebuilt from source to even try any of these options out.
Our current "best" practice is using Helm + Terraform, bootstrapping secrets with Terraform Cloud, and ensuring all services run in their own isolated namespace and service account. This feels inadequate.
At this point, I really have no idea how people are using secrets in the wild.
What
"A data field is used to store arbitrary data, which is encoded with base64 so that no one can view the values of secrets."
That being said, I don’t really use kubernetes secrets except to configure applications that expect them. They are a pain to manage. For most things it is better to use your cloud’s built-in secret manager or something like Vault (however Vault is non-trivial to set up correctly).
I found happiness by either using AWS KMS (I assume other clouds offerings can do the same) by then storing the encrypted blob in K8s Secrets or through the Vault sidecar approach (obviously when using Vault only).
It's sad but undeniable anecdote, most easily explained by ignorance; Red Hat is just an IBM brand serviced by low skill/cost IT drones now.
The fact that Kubernetes uses base64 for storing secrets by default is another head shaker. One would think self respect would preclude this sort of thing.
Edit: There is reason to do it explained above[1] - to store binary data in the yaml, but doesn't change fact you don't have to use it.
[0] https://kubernetes.io/docs/concepts/configuration/secret/#ba...
Yes it is.
Caution:
Kubernetes Secrets are, by default, stored as unencrypted base64-encoded strings.
Citation requested.
Whole podcast: https://share.transistor.fm/s/8cf33e94
Secret management section: https://overcast.fm/+Xq7xA2hYA/42:00