Google Cloud’s Secrets Manager

Hi Seth!

I wonder how is GCP Secrets Manager comparing to Hashicorp Vault? Is GCP SM implemented in Hashicorp Vault (or Berglas) behind the scene? If not, what are the differences in terms of feature set?

Hey Seth, I know there's been _quite_ the effort under the hood to make this product a reality and am curious if you would be open to writing a bit about lessons learned to help others learn about the research & development process.

Your vault-on-gke[0] repo is solid. For greenfield GKE projects starting now, would you recommend Secret Manager or vault-on-gke?

0: https://github.com/sethvargo/vault-on-gke

Seems more comparable to parameter store, which comes in both a paid and "free" version. Secrets Manager does have some of the same capabilities, but it also extends it to things like short lived credentials.

For AWS, as with many cloud offerings, read the pricing small print: AWS Secret Manager also charges per API request [1]. It isn't expensive, but you should keep it in mind when you architect your infrastructure. (We actually switched from fetching secrets at runtime to injecting them into containers are deploy time, and this was one reason).

https://aws.amazon.com/secrets-manager/pricing/

you can store gcp secrets, in multiple regions, so you pay for every region 0.06 cent, i.e. if you only want one region you only pay 0.06 cent

Sorry the pricing is unclear. In Secret Manager, secrets are versioned (a "secret" is a named collection of "secret versions"). We only charge for active secret versions (you can destroy secret versions, usually after a rotation operation).

The secret version contains that actual secret data (i.e. "ABCD1234"), and you can choose the regions in which you want that secret data replicated. Each region you choose is $0.06.

So if you had 1 secret with 12 versions stored in 2 regions, that would be 12 x 2 x $0.06/mo = $1.44. Hope that helps!

EDIT: replaced "*" with "x" in math because it was getting parsed as italics

Enable Cloud Audit Logging for Secret Manager and give us feedback. We know auditing is super important, so it's built in!

Have you looked at just using AWS parameter store?

Hashicorp vault is basically industry standard I believe. But there are various cloud APIs too, which have varying degrees of integration with kubernetes.

Azure Keyvault: https://docs.microsoft.com/en-us/azure/key-vault/key-vault-o...

I'm using the SSM Parameter Store [0] a lot, because of its easy integration in applications – many services can directly pull secrets from it and provide them as environment variables. The secrets can be KMS encrypted and cost a fraction of the Secrets Manager, which I think is just the parameter store with secrets being rotated by a Lambda.

[0] https://docs.aws.amazon.com/systems-manager/latest/userguide...

AWS Secrets Manager is fine if you're in that cloud. We inject all our secrets into the API containers' environments when they're deployed. For some things that have refresh tokens and access tokens with shorter lifetimes, we use the Secrets Manager API to fetch them when necessary at runtime.

Salt+gpg works, although there are a lot of assumptions baked into that.

If you have an enterprise, HashiCorp Vault has the best features and integrations that I know of (everyone should be using short lived credentials, but who actually does if they're not using Vault?). If you can't use that, AWS Secrets Manager for its simpler integration of secret rotation, and if that's too expensive, AWS SSM PS. Google Cloud Secrets Manager would be analogous to AWS Secrets Manager.

The benefit of AWS Secrets Manager has been that it integrates directly with other services. In ECS/Fargate, you can create a task definition and list secrets to be injected at run time. It also has features like secret rotation so you don't have to roll your own (well, you kinda do w/Lambda, but they have samples), and you can use things like IAM to lock down access per-secret or per-IAM-ARN-hierarchy.

Like everything in AWS, it'll cost you more, but it'll also eliminate a lot of engineering time. If you're trying to pinch pennies, use Vault or something hokey like pulling a token from AWS SSM PS or S3 and rely on IAM roles/profiles/etc for access control.

(shameless plug as well) Try out CyberArk DAP (formerly Conjur): https://hub.docker.com/r/cyberark/conjur. OSS, powerful RBAC, has k8s-native workload ident auth, works on OCP and w/ PCF, lots of tooling integrations (https://github.com/cyberark/summon, https://github.com/cyberark/secretless-broker, etc), and there's a relatively straightforward path up to "enterprise"-y stuff if you think you want the bells and whistles (replication, auto-deployable followers, multi-zone failover, etc).

As others have mentioned, also using SSM Parameter Store (if you're in AWS) + KMS. Running most of our micro services using ECS (Fargate) which can populate environment variables directly from secrets in SSM Parameter Store.

Vault can do a lot more than AWS Secrets Manager -- possibly GCP too but I've not checked. So if you're paying for Vault (bare in mind Hashicorp also do a free version) then odds are you want enterprise features rather than an encrypted key/value store.

Vault does more than store secrets.

For example, with Postgres it can store root creds and issue non-root, time-bound creds for each service. Vault's "backends" allow it to do _a lot_ of things that a secure static credentials manager can't.

Not to say that this isn't good though – any proper secret management system is better than none :D

While I cannot comment on specifics, we are going to provide deeper, first-class integrations with other GCP products.

while a bit of a toy still, might be of interest: https://github.com/masonwr/CloudSecret. basic CRD that maps google "managed" secrets to Kubernetes secrets.

Me too!

> Can I only give access to a specific secret to a specific person/group of people?

yes.

> Playing with Secrets Manager it looks like anyone with access to the project can access all secrets.

just FYI, this is not the case. the "Secret Manager Secret Accessor" role must be applied per-secret for any identity to read the content of that secret.

KMS manages encryption keys. A secrets manager gives you an API for storing and referencing arbitrary named secret values with audit logs, etc. for them, which are encrypted using KMS.

For example, this might be something like “store a secret ‘my app remote API key’ encrypted with a regularly rotated key and allow ‘my app container’ to retrieve the value”.

The short answer is KMS doesn't store secrets.

You can use KMS as part of your own secret manager, but you would have to store the actual secrets somewhere else, like GCS or Datastore. KMS stores encryption keys, and has an API for encrypting/decrypting. For example to retrieve a secret using KMS you would get the encrypted data from where it is stored, like Datastore, and send it to KMS to have KMS decrypt the data. Secret Manager actually stores the secret, so a single API call can retrieve the decrypted value. Secret Manager also has versioning, which is important when rotating secrets. If you were building your own solution around KMS you would need to do versioning in your storage schema, wherever you end up storing the secrets.

Me too. Everyone else calls key storage "Secrets Manager" (plural)