Why We Need Dynamic Secrets
hashicorp.com
hashicorp.com
Should an individual leave an organisation then the credentials they have obtained from Vault to access a datastore would expire automatically, normal process would apply to remove them from LDAP and disable the ability to require further credentials.
There is always a process problem with managing secrets but dynamic secrets in Vault stops long-lived secrets and reduces unofficial password sharing.
The goal of Vault was to be a modern Kerberos, but invert the integration model. I think that is the Achilles heel of Kerberos, since it has a complex API and only works if the endpoint systems are tightly integrated.
Vault operates in much the same way, and could be viewed as a "KDC". However, instead of requiring the Authentication Service (AS) to be Vault aware, Vault uses authentication plugins to do the integration in the other direction. Similarly, instead of network services being Vault aware, Vault uses secret plugins to do the integration with endpoint systems.
This lets Vault easily be extended to support new authentication systems and endpoint systems, without needing those systems to be modified. Otherwise, its conceptually very similar to Kerberos!
Edit: I'm a co-founder of HashiCorp, and one of the early authors on Vault.
I would caveat, however, that encryption is just one part of the tool in the toolbox. A correct security posture is about having layers of defense. The next layer is absolutely the application and isolating secret management responsibility from the app layer to the infrastructure layer is the right strategy, IMO.
Very chicken and egg.
In some scenarios it’s possible to at least avoid introducing another vault-specific credential. For example if your workload is running in a managed environment like AWS or Kubernetes, the platform provides a credential automatically (IAM role or service account). If the vault can recognize this platform credential and map it to a role defined in the vault, then you are using the platform credential as the “secret 0”.
It’s still not completely idiot proof because you have to define the mapping between platform roles and vault roles, and define permissions between vault roles and secrets.
And of course the detailed workflows for doing this vary between vaults.
In the automated case, Hashicorp Nomad can issue tokens to individual applications. I was working on that but left the company before getting it into production. That does require trusting your Nomad infrastructure with long-term tokens, but that's still quite a bit better than, e.g., committing tokens into Git :)
Effectively, there are only two possible approaches. Platform specific integrations, such as AWS IAM, Kubernetes, Nomad, etc. Or Trusted Orchestrators, which help inject the initial "secret zero".
Under the 'trusted orchestrator' model the article you linked to describes it states "you have an orchestrator which is already authenticated against Vault with privileged permissions".
https://www.vaultproject.io/guides/identity/secure-intro.htm...
In the case of the AppRole auth method and where the orchestrator is an automated app, would it be fair to suggest that you'd have an AppRole for which secret IDs are generated that do not expire based on time and/or max uses?
From the Vault documentation here: https://www.vaultproject.io/api/auth/approle/index.html#crea...
Is it possible to set 'secret_id_ttl' to something like '0' so that it never expires?
The reason behind my question is that if the automated app is initially seeded with a role and secret ID it can login to Vault and get a token, which can be renewed going forward (based on settings for the AppRole). However, if the token is not renewed in time, or the service has to be restarted, you would need to regenerate the secret ID and reseed the application with it, which would be a manual process.
Thanks for any advice :)
Exactly what you suggested would work! Having an AppRole that never expires would allow the trusted orchestrator to authenticate on each run, and then generate and inject ephemeral credentials.
Solving the human trust problem is far more difficult and outside the realm of a single software application.
"But... there's a root account... creating another account... which is typically a privileged action. What's protecting that? Is that root account being rotated, too?"
I'm talking about the account creating the database user. Let's take MSSQL, for example. The equivalent to a root account there is `sa`. So, Vault will have control of the `sa` account in order to create leased database users.
If I'm a malicious actor inside the environment, what's stopping me from compromising the `sa` account and mimicking dynamic secrets? I'd need to be comparing Vault logs with Database logs constantly to ensure it was legit.
It's just madness, if you ask me.
You can do a lot to minimize that, many DB auth systems let you specify where connections for users are allowed to come from, only allow a single concurrent connection, etc. Plus you can encrypt and verify that connection with TLS, build up FW rules, etc.
Security is never "perfect" but to say dynamic secrets are wrong and crazy just because of this one problem isn't very well thought out. Your wordpress application will be hacked a LOT easier than your MSSQL sa account.
So dynamic secret people are saying, we all know our apps leak/get compromised all the time, minimize their leakage with a short lived, one-time use secret, that can only be used from X machine, etc, instead of a long-term, static secret that can be used anywhere, etc.
For sure, for high-value targets you best be auditing everything you care about, not just vault, but with a vault audit log, you can automate a lot of that comparison between MSSQL and vault, where before it was much harder to automate since the logs from every application using MSSQL will be different, etc. But if you are in a high-value org like this, you are almost certainly already auditing your MSSQL account creations, this doesn't change that.
At a smaller scale, we've been satisfied with encrypting secrets using KMS and then placing the resultant ciphertext into environment variables that we commit with our terraform scripts. Our system does not allow granular access control, but it is relatively simple to implement using a couple of Bash scripts and allows committing all of the config, including secrets, all at once so that deploys are fully reproducible.
To allow a bit more granularity in isolating different environments, we use different encryption keys depending on whether we have a dev/uat/prod deployment. We grant the target application role access to particular secrets based on which KMS keys it can use.
The other trick is that we decrypt all of the secrets in our environment at the entry point of our application code, so that the environment is fully decrypted by the time the service runs. This means there is no trace of secret management in our application code.
There are intermediate steps of this scheme where we could revoke access to KMS keys for some developers, so that we could allow users to deploy services but not necessarily be able to use and access the encrypted secrets.
Regardless, I always appreciate knowing these kinds of technologies exist because they seem to me to solve very challenging problems at scale.
In short, I think dynamic secrets will increase the threat surface dramatically, while simultaneously increasing complexity. Complex systems are expensive to maintain, hard to secure and notoriously fragile. There is a limited scope use case where dynamic secrets may be a good fit, but I for one wouldn't base a purchasing decision on whether or not my secret store can do dynamic secrets. Instead, I'd want a secret store that has some provision for secrets rotation that I can tailor to the needs of my specific use cases.
In terms of the "compute cost", for an infrastructure of that size this is a negligible amount of overhead. For dynamic secrets that live 30 days, rotating 500K secrets works out to 1 secret every 5 seconds.
The advantage would be avoiding an incredible number of static credentials sprawled across a very large estate, plus having a unique audit trail that lets you identify points of compromise. Treating those credentials as dynamic will also reduce the human overhead of managing so many credentials, instead focusing on roles and high level intents.
I question if there is an non-disclosed bias given the anonymous user, created just in advance of the comment.
Like I said, I think there is a use case here for dynamic secrets, but I have questions about what it looks like when it comes to trying to do them at scale. If you have solutions to the worries I outlined, I'd love to hear them.
We actually deploy vault tokens to every user(happens at login time, since login authentication happens against vault). So we can deploy secrets directly to end users using vault policy.
Applications that are not ran/started directly by the user find other means of getting their secret 0(VAULT_TOKEN), usually through nomad, but in the rare case it's a standalone application not run under nomad, we seed the startup with a token by hand, and then have a process handle renewals, etc.
For instance: Vagrant Images/boxes hosted by hashicorp (vagrantcloud) are neither signed nor is there any author information available (if not explicitly provided by the uploader).
I believe that using Dynamic Secrets is HashiCorp's proposal for how to mitigate this; leave the secrets in the log files, but make sure they expire in a timely manner.
How the heck can I know if it was a legit created database user or not?
Vault typically prefixes something to the username as well (e.g. "vault-...") and also audits the creation of dynamic users so you can either look for the prefix or cross check the audit logs.
But based on what you've just asked, I'd definitely never create anything like what you're working with because it sounds like ANY addition to your infrastructure is problematic.
So whatever is accepting the secret must be checking with Vault to verify the secret.