The Current Secrets Rotation Process Is Broken | Hacker News Reader

The Current Secrets Rotation Process Is Broken

doppler.com

68 pointstompic82321 comments

What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking.

But secrets are next to useless if they are:

- not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys)

- not regularly rotated (at the very least when a person that had access to them leaves the company)

And rotation is hard: a lot of systems still don't support multiple keys, so rotation has to be very carefully tied to some form of blue-green deployment, which is often not possible.

So much practice around secrets and identity is oriented around assuming secrets are expensive. Like, people will set up a database and create ‘the database user’, with its one password which now needs to be used by all the systems that access that database.

To maximize security you need to treat secrets as being cheap. In fact they need to be disposable.

In theory any time a system needs access to that database you could have a process that creates an entirely new user just for that session, with a random password - and, bonus, only the permissions that that particular client needs.

If that secret gets compromised, no biggie - you can destroy that user account and the credential will never work again.

Secret rotation is a related approach, but again - if your rotation frequency is ‘quarterly’ rather than ‘hourly’, is there a good reason for that, or have you again just assumed secrets are expensive?

I agree with the idea of cheap, disposable, fine-grained secrets.

With databases in particular the problem is that a connection is a relatively expensive thing which takes noticeable time to establish. Hence connection pooling, and the need for all sessions of a particular class to share credentials.

Right, secrets are a means to an end: giving one entity access to another.

> To maximize security you need to treat secrets as being cheap. In fact they need to be disposable.

Maybe even take it one step further: secrets are invisible implementation detail.

It does come with it's own challenges, of course, and there are no widely adopted standards. You can use something like IAM within AWS, but IAM can't give your apps access to Stripe, for example.

> In theory any time a system needs access to that database you could have a process that creates an entirely new user just for that session, with a random password

But doesn't that process just become the new login method? Doesn't that process need its own secrets to have the (much greater) authority to make new users? Doesn't compromising this process give you much more power than compromising one system would have?

Right, secret management schemes do have a tendency to feel like ‘but wait, who holds the keys to get hold of the key?’ - it has to be turtles all the way down, right?

Except it turns out not. Secret issuance schemes like this let you move trust boundaries around, they don’t absolutely require all powerful admin systems to exist, and they also let you layer secret schemes on top of more robust trust relationship systems like IAM.

One common pattern is that higher trust ‘control plane’ systems create ephemeral lower trust process instances (eg for autoscaling) - and so patterns like this allow those control planes to bootstrap the child systems with their own unique, cheap, minimally permissioned credentials.

Yes the control plane is a juicier higher priv target. But that’s already the case.

>And rotation is hard: a lot of systems still don't support multiple keys, so rotation has to be very carefully tied to some form of blue-green deployment

Oh just today I found out the previous team stored all secrets in git in multiple repositories, and now I need to rotate them all. They encrypted them eventually with sops but did not rotate them so you can find everything in git history. The only solution I can come up with so far is create new credentials and gradually migrate everyone to new secrets, and then delete the old ones. And there's still a chance we can forget something and something will break. I wish there was a button "rotate secrets"...

Hey, maybe try running a GitGuardian [1] scan on all those repositories to look for hardcoded secrets. GitGuardian can also test in some cases if the secrets are valid or not, meaning you have to revoke and rotate them asap. I hope this helps.

[1] https://www.gitguardian.com/monitor-internal-repositories-fo...

Disclaimer: I work for GitGuardian.

I find it frustrating that people still write programs without thinking about secrets rotation. It's kind of understandable that older stuff didn't think about it, since there was certainly a time when people treated secrets, especially things like API keys, as effectively indefinite, but that just doesn't make sense, especially not now. Same for any ephemeral keymatter, or other cryptographic keymatter: there should be a way to have multiple, with at least one "primary" secret and at least one secondary secret. The way I often do this is as simple as having an environment variable/flag/whatever for two separate values and try them in order. Rotating secrets used to salt or encrypt database entries is definitely a bigger pain in the ass, but if it's a big pain in the ass to design a system for it when you're building out a feature, you'd better bet it will be a pain in the ass when you've got a huge security breach and are faced with the task to do it not only in short order, but in concert with many other tasks related to the ongoing fire.

There are definitely other things to be said about the state of secret tokens and secret management, but this one is honestly not that hard to do significantly better at. Rotating secrets should not involve downtime, it should be something you can do out of precaution...

I get what you're saying, but let's not downplay how difficult secret rotation is out the the real world of physical devices.

I work on robots. Let's say you want to rotate the very important image signing keys. If you're lucky enough to have hardware that supports multiple keys (not universal), your process might only be as painful as producing doubly-signed images, coordinating global updates, and field servicing an inevitable percentage of the fleet that had weird issues.

Worst case, you'd have to find every unit around the world and physically replace the PCBs.

I'm personally going to avoid both without some extremely good reasons.

Yep, physical devices are certainly a worst case scenario depending on the constraints. I'm purely talking about software running on servers or, at worst, at least end user computers.

The other day we were sketching out how to integrate another team’s service into ours. They wanted us to compile their API key into our application (in CI/at build time).

Believe it or not, I've absolutely been there. It blows my mind.

At LEAST let me set the key at runtime. When it comes to stuff that I ship to end users, I'd prefer to be able to dynamically grab the API keys from an endpoint I control for various reasons anyways...

You're exactly right about those two problems, and they actually go together quite well. If a user who previously had access to a secret suddenly has that access revoked, you should rotate that secret. Technically that's only really necessary if the user ever actually saw that secret value, but most secret managers don't expose that information.

At Doppler, we're currently in the process of building out our rotation engine[0]. It allows you to automatically rotate secrets with third party providers (including your database) on a schedule you define. We also provide access logs to see exactly who saw what and when. The ultimate goal is to automatically rotate your secrets as users' access changes. And if you get notification of a breach, you can rotate everything with a single click.

[0] https://docs.doppler.com/docs/secrets-rotation

The last part is the key thing. I'm in a continuous running battle with our information security risk team over this - they believe that secret rotation is a secrets management platform feature (and we use Vault so that box is ticked) but the reality is that secret rotation that isn't tied to restarts / reloads of said secrets within distributed application estates is just another way to have an incident.

This is particularly true if you've backed into secrets management as a practice and retrofitted existing platforms; but even if you've designed net new, it's an abstraction-breaker to get reloads of credentials happening all the way down the stack.

Even a blue/green deployment isn't a panacea if there're databases or other shared states that depend on the credential cutover.

From my point of view, the better approach is not to share secrets at all where possible and drive everything from service identity directly. e.g. short-lived credentials like AWS IAM or short-lived PKI proofs like SPIFFE.

The "better way" for me has been managed identities in Azure. These are Azure Active Directory service principals, which are similar to Group-Managed Service Accounts (gMSA) in Active Directory. They're essentially a platform-managed identity with automatically rotating secrets that the developers never need to know, and sysops don't have to deal with. They can be assigned to VMs or various PaaS services, and then service-to-service access is via standard Azure RBAC, with auditing and everything.

There is nothing to rotate, nothing to put into a "key vault", nothing to worry about leaking out, nothing that could accidentally get pasted into Git, and nothing on that developers' laptop that he left at the airport. You literally say: "I want service A to have read access to service B", and you're done.

Visual Studio automatically injects the developer's personal identity into processes, so that they can access cloud resources during local development, so that's covered too. (For Linux developers, the Azure Identity SDK can pick up the user credentials via the "az" cli.)

"Making fire with stones to avoid a proprietary lighter." -- /user?id=pjmlp

I'm having the same success with Google IAM, and I agree it completely solves the problem, except for one thing; It only works with compatible services, within your company boundary, within the same cloud provider whereas API keys or oauth flows work everywhere.

The fact that we don't have a well adopted standard for password management, critically including password rotation and key sharing, by now is both surprising and infuriating.

Are there any widely available, published threat models for secret management of this sort? It seems to me that if you want to frame the discussion in a useful way, you'd start there. Then you can point out where current practice works, where it doesn't, and how your solution works better.

HashiCorp vault is a pretty good answer. It has plugins to auto rotate a number of different connections, works with both static and dynamicly created users, whole host of auditing optinons.

Doppler looks interesting - are there other companies in this space doing good work?

Reply on news.ycombinator.com