Hacker News Reader

Comment by pjbster | Hacker News Reader

The best approach I've seen is to have the secrets in a vault and to keep the vault URL in an env var (so different vaults can be used for dev and prod, for example). The secrets location within the vault is held in a variable defined in an application config file which is checked in to source control. At runtime, a piece of code consults the variable and queries the vault using the provided location and then dynamically configures the db connection with the returned credentials. The credentials are dumped from memory once the transaction completes.

(I should also point out that this is in a batch ETL system.)

The nice thing about this approach is that it is virtually impossible for secrets to end up in source control. The risk is that the vault becomes a major single point of failure.

View on HN

It also makes cycling credentials easy. All you have to do is update the values in the vault. No need to hunt down all those places where you set environment variables.

I think you don’t even need to redeploy any long-running tasks. If they have a connection open, it should continue working, and if they don’t, they should consult the vault when they open one and get the new credentials from it (this may be different for different kinds of credentials)

There will be a race condition between creating credentials and storing them in the vault, but even that can be avoided:

  - create new credentials with same rights as the old ones
  - update vault
  - delete old credentials

Another race where credentials get changed between reading them from the vault and using them can’t be avoided, but if you wait a few minutes before you delete the old credentials, it’s extremely unlikely you’ll hit it.

How does the system authenticate with the vault? Why not use that same system to authenticate the database connection? A vault is only useful at scale or with additional compliance requirements. Otherwise, keep it simple. Very few systems actually need that additional level of indirection.

I don't know and I don't need to know (I'm a mere dev -- no "ops" in my role). And, yes, this client has very high compliance requirements.

There are often multiple sets of credentials that you need to pass to a microservice, some of which may be shared between multiple instances of the service or even between multiple services. Changing them would require plenty of case-specific updates to service configurations or just one update in the vault. By reducing the amount of work to update the credentials you also reduce security and quality-related risks.

It's usually rely on some globa mechanims by the underlying architecture.

Example, if it's AWS it may rely on ec2 instance role, to allow it access to the secret manager.

If it's kubernetes, it can be done through k8s token mount, basically allow token in namespace access the vault, and the token(which is generated and manage by k8s, which is just a JWT btw) is mounted into your pod.

Do you have any particular vault in mind? We are looking for a simple solution which would work exactly like you described. We just need couple of different sets of variables for our web server, workers and other services.

I'm not sure which vault we're using. There's no identifying information in the server responses. The storage type shows as "Consul" so that suggests Hashicorp Vault.

I'm not the person to ask how to set it up but it's seamless in use and can return data in json format which makes it easy to parse. If it were my money I'd look very hard at this product.

I've used GCP secrets manager and it worked well. Pretty simple API and the option for providing your own encryption keys if you want it.

I built onboardbase.com for this exact purpose and it has continuously solved a lot of use cases for me personally. There is also hashicorp vault which is amazing as well.

This is the way!

Reply on news.ycombinator.com