Hacker News Reader

Comment by hbrn | Hacker News Reader

What's sad is that despite it's current state, secret management has still managed to turn into a cargo cult. It's a "best practice" that people blindly implement without thinking.

But secrets are next to useless if they are:

- not used to limit number of people that have access to them (it is quite typical in small teams to give everybody access to production, which essentially gives you access to keys)

- not regularly rotated (at the very least when a person that had access to them leaves the company)

And rotation is hard: a lot of systems still don't support multiple keys, so rotation has to be very carefully tied to some form of blue-green deployment, which is often not possible.

View on HN

So much practice around secrets and identity is oriented around assuming secrets are expensive. Like, people will set up a database and create ‘the database user’, with its one password which now needs to be used by all the systems that access that database.

To maximize security you need to treat secrets as being cheap. In fact they need to be disposable.

In theory any time a system needs access to that database you could have a process that creates an entirely new user just for that session, with a random password - and, bonus, only the permissions that that particular client needs.

If that secret gets compromised, no biggie - you can destroy that user account and the credential will never work again.

Secret rotation is a related approach, but again - if your rotation frequency is ‘quarterly’ rather than ‘hourly’, is there a good reason for that, or have you again just assumed secrets are expensive?

I agree with the idea of cheap, disposable, fine-grained secrets.

With databases in particular the problem is that a connection is a relatively expensive thing which takes noticeable time to establish. Hence connection pooling, and the need for all sessions of a particular class to share credentials.

> In theory any time a system needs access to that database you could have a process that creates an entirely new user just for that session, with a random password

But doesn't that process just become the new login method? Doesn't that process need its own secrets to have the (much greater) authority to make new users? Doesn't compromising this process give you much more power than compromising one system would have?

Right, secret management schemes do have a tendency to feel like ‘but wait, who holds the keys to get hold of the key?’ - it has to be turtles all the way down, right?

Except it turns out not. Secret issuance schemes like this let you move trust boundaries around, they don’t absolutely require all powerful admin systems to exist, and they also let you layer secret schemes on top of more robust trust relationship systems like IAM.

One common pattern is that higher trust ‘control plane’ systems create ephemeral lower trust process instances (eg for autoscaling) - and so patterns like this allow those control planes to bootstrap the child systems with their own unique, cheap, minimally permissioned credentials.

Yes the control plane is a juicier higher priv target. But that’s already the case.

Right, secrets are a means to an end: giving one entity access to another.

> To maximize security you need to treat secrets as being cheap. In fact they need to be disposable.

Maybe even take it one step further: secrets are invisible implementation detail.

It does come with it's own challenges, of course, and there are no widely adopted standards. You can use something like IAM within AWS, but IAM can't give your apps access to Stripe, for example.

>And rotation is hard: a lot of systems still don't support multiple keys, so rotation has to be very carefully tied to some form of blue-green deployment

Oh just today I found out the previous team stored all secrets in git in multiple repositories, and now I need to rotate them all. They encrypted them eventually with sops but did not rotate them so you can find everything in git history. The only solution I can come up with so far is create new credentials and gradually migrate everyone to new secrets, and then delete the old ones. And there's still a chance we can forget something and something will break. I wish there was a button "rotate secrets"...

Hey, maybe try running a GitGuardian [1] scan on all those repositories to look for hardcoded secrets. GitGuardian can also test in some cases if the secrets are valid or not, meaning you have to revoke and rotate them asap. I hope this helps.

[1] https://www.gitguardian.com/monitor-internal-repositories-fo...

Disclaimer: I work for GitGuardian.

I find it frustrating that people still write programs without thinking about secrets rotation. It's kind of understandable that older stuff didn't think about it, since there was certainly a time when people treated secrets, especially things like API keys, as effectively indefinite, but that just doesn't make sense, especially not now. Same for any ephemeral keymatter, or other cryptographic keymatter: there should be a way to have multiple, with at least one "primary" secret and at least one secondary secret. The way I often do this is as simple as having an environment variable/flag/whatever for two separate values and try them in order. Rotating secrets used to salt or encrypt database entries is definitely a bigger pain in the ass, but if it's a big pain in the ass to design a system for it when you're building out a feature, you'd better bet it will be a pain in the ass when you've got a huge security breach and are faced with the task to do it not only in short order, but in concert with many other tasks related to the ongoing fire.

There are definitely other things to be said about the state of secret tokens and secret management, but this one is honestly not that hard to do significantly better at. Rotating secrets should not involve downtime, it should be something you can do out of precaution...

The other day we were sketching out how to integrate another team’s service into ours. They wanted us to compile their API key into our application (in CI/at build time).

Believe it or not, I've absolutely been there. It blows my mind.

At LEAST let me set the key at runtime. When it comes to stuff that I ship to end users, I'd prefer to be able to dynamically grab the API keys from an endpoint I control for various reasons anyways...

I get what you're saying, but let's not downplay how difficult secret rotation is out the the real world of physical devices.

I work on robots. Let's say you want to rotate the very important image signing keys. If you're lucky enough to have hardware that supports multiple keys (not universal), your process might only be as painful as producing doubly-signed images, coordinating global updates, and field servicing an inevitable percentage of the fleet that had weird issues.

Worst case, you'd have to find every unit around the world and physically replace the PCBs.

I'm personally going to avoid both without some extremely good reasons.

Yep, physical devices are certainly a worst case scenario depending on the constraints. I'm purely talking about software running on servers or, at worst, at least end user computers.

You're exactly right about those two problems, and they actually go together quite well. If a user who previously had access to a secret suddenly has that access revoked, you should rotate that secret. Technically that's only really necessary if the user ever actually saw that secret value, but most secret managers don't expose that information.

At Doppler, we're currently in the process of building out our rotation engine[0]. It allows you to automatically rotate secrets with third party providers (including your database) on a schedule you define. We also provide access logs to see exactly who saw what and when. The ultimate goal is to automatically rotate your secrets as users' access changes. And if you get notification of a breach, you can rotate everything with a single click.

[0] https://docs.doppler.com/docs/secrets-rotation

The last part is the key thing. I'm in a continuous running battle with our information security risk team over this - they believe that secret rotation is a secrets management platform feature (and we use Vault so that box is ticked) but the reality is that secret rotation that isn't tied to restarts / reloads of said secrets within distributed application estates is just another way to have an incident.

This is particularly true if you've backed into secrets management as a practice and retrofitted existing platforms; but even if you've designed net new, it's an abstraction-breaker to get reloads of credentials happening all the way down the stack.

Even a blue/green deployment isn't a panacea if there're databases or other shared states that depend on the credential cutover.

From my point of view, the better approach is not to share secrets at all where possible and drive everything from service identity directly. e.g. short-lived credentials like AWS IAM or short-lived PKI proofs like SPIFFE.

Reply on news.ycombinator.com