Hacker News Reader

Comment by JadoJodo | Hacker News Reader

> Don't you feel some anxiety given we've normalized committing encrypted secrets to git repos?

Maybe I haven't worked at enough places, but... when has this ever been allowed/encouraged/normalized?

You'd be surprised. In the past I was on a big project at company with multi-billion $ revenue. They got caught with their pants down on an audit once because people would not only commit credentials into internal repositories, they were usually not encrypted at all, among other deeper issues. It sparked a multi-year long project of incorporating a secrets management service into the 1000+ repositories and services the company used. Found a loooooot of dead bodies, tons of people got fired during the process. After that experience I imagine this practice is fairly common - people, even smart developers, don't always seem to be able to comprehend the blast radius of some of these things.

One of my favorite incidents during this clean-up effort was, the security team + my team had discovered a lot of DB credentials were just sitting on developer's local machines and basically nowhere else that made any kind of sense, and they'd hand them around as needed via email or message. So, we made tickets everywhere we found instances of this to migrate to the secret management platform. One lead developer with a privileged DB credential wrote a ticket that was basically:

"Migrate secret to secret management platform" and in the info section, wrote the plaintext value of the key, inadvertently giving anyone with Jira read access to a sensitive production database. Even when it was explained to him I could tell he didn't really understand fully why that was silly. Why did he have it in the first place is a natural followup question, but these situations don't happen in a vacuum, there's usually a lot of other dumb stuff happening to even allow such a situation to unfold.

> Found a loooooot of dead bodies, tons of people got fired during the process.

I'm genuinely curious as to what the fireable offenses here would be. If the company had an existing (broken) culture of keeping unencrypted secrets I wouldn't expect people following that culture to be fired for it.

Okay, but that sounds like a very different situation than a small shop where encrypted secrets are committed to one file per-repo, and keys and secrets are rotated regularly.

Okay, in case it was missed, my salient point was that this behavior is very common and provided a ridiculous example as my evidence. I'm making no commentary on the practice itself (although I do think committing configs like secrets is really silly and anti-productive)

You think tracking configuration in source control is anti-productive?

When it comes to secrets, usually yes, as my post indicated. YMMV

You said "like secrets". I wasn't sure what that modifier extended to exactly...

What's wrong with committing encrypted secrets? That's how I use `sops`.

You can’t revoke, rotate, or audit access to them.

You can rotate them, although admittedly it can be more or less complicated depending on how your tooling compares to that of the secrets management system you are comparing against.

You can't necessarily revoke a secret just because it is in Hashicorp Vault or AWS Secrets Manager. Revocation is a function of the system that provisions and/or uses the secret for authentication, not the system that stores the secret. E.g. if you generate a certificate and store the private key with vault or sops, the revocation procedure is identical and has nothing to do with the secrets storage system.

Auditing access can be done coarsely by auditing access to the encryption key. Admittedly, this is an area where a more sophisticated system offers benefits. Although it isn't exactly iron clad -- a service may access a secret then leak or transfer the secret without that being visible in the audit log.

I would venture to guess the main concern is accidental commit of decrypted secrets.

That can happen regardless.

If a key gets compromised, the encrypted secrets are compromised forever, since you can't be sure all the git clones everywhere can be updated with a new encryption key. Not to mention how fiddly it is to edit git history.

I would assume if you are committing encrypted secrets you would make sure they are rotatable

But you can and should be rotating those secrets on some schedule regardless, and if you find out a key has been compromised you can immediately rotate the secrets.

Wait, why are there so many skeptics in this thread?

I have setup AWS + SOPS in several projects now, and the developers do not have access to the secrets themselves nor the encryption key (which is stored in AWS). Only once did we ever require to rollback a secret and that happened at AWS level, not the code’s. Also it happened within the key rotation period, so it was easy.

For us it’s easier to track changes (not the value, but when it changes), easier to associate it with incidents.

Indeed, the only time I saw this was a decade ago for a temporary POC... not doing this is a good defense-in-depth practice even if the encryption is solid.

Reply on news.ycombinator.com