Kubernetes External Secrets

Kubernetes External Secrets essentially copies secrets from an external source, in this case AWS Secrets Manager, and creates a native Kubernetes secret in the same namespace where the ExternalSecret custom object was defined.

Kubernetes is being leveraged as a secrets cache and transport layer, with just enough access control thanks to the built-in RBAC functionality. Kubernetes is not the source of truth for the contents of the secret, but provides APIs for referencing, and ultimately transmitting cached secrets to containers under management.

I like this approach. Nice and simple. I do have one suggestion that might improve External Secrets for use with multiple clusters.

By running the External Secrets "control plane" inside the Kubernetes cluster you end up with what I described above. Now consider the case where you have multiple Kubernetes clusters in the same environment -- HA across multiple regions in production.

You have a few options on how to leverage External Secrets in this context. The obvious approach is to deploy the "control plane" across every cluster. There are pros and cons. While you have the ability to do a canary roll out of the External Secrets control plane across multiple clusters, you also take up compute resource across every cluster, and must store the credentials required for the External Secrets control plane to sync secrets from an external store.

The other option would be to enable the External Secrets control plane to live outside of any individual Kubernetes cluster and work as a "global control plane". In this configuration the External Secrets control plane can push secrets across multiple clusters. Users can still define ExternalSecrets objects in each cluster, but you now have the ability to support "global" ExternalSecrets objects which are replicated to every cluster, or a limited set of clusters and namespaces based on configuration.

You can still host the External Secrets control plane using Kubernetes, but in a "admin" cluster, which is separate from the clusters where normal applications are deployed.

We're building something similar at work. There is actually an implementation of the very same mechanism using the operator framework. Here's the link: https://github.com/mumoshu/aws-secret-operator

Were you aware of this project? If so, why did you go for this implementation?