How to mitigate risk from secrets leaks | Hacker News Reader

How to mitigate risk from secrets leaks

reversinglabs.com

91 pointsvmatsiiako55 comments

You can't leak API keys if there are no API keys to leak! The article recommends OIDC for apps, which is a step up, especially if you rotate the bearer token, however there is another option - use short-lived certs.

Our project Machine ID is replacing API keys with short-lived certificates:

https://goteleport.com/docs/machine-id/introduction/

Another great option is SPIFFEE https://spiffe.io/

The adoption is slower than we wanted, because it's not trivial to replace API keys, but we see more and more companies using mTLS + short lived certs as alternative to shared secrets.

How does this approach practically differ from using short-lived JWTs+TLS?

Short version is that with mTLS and short-lived certificates you don't have to worry about anyone stealing and re-using your JWT tokens and revoking tokens.

LVH from Latacora explains it way better than I could in "A child's garden of inter-service authentication" [1]

However, here is my view:

If your token is not bound to the connection, someone can steal and reuse it, just like any other token. It is possible to use OAuth token binding [2], but at this level of complexity, mTLS + short lived certs deliver the same security and are easier to deploy.

It's easy to mess up JWT signatures, although, to be fair, it's not like X.509 certificates format is any better, however it's been more tested over years of use.

[1] https://latacora.micro.blog/2018/06/12/a-childs-garden.html [2] https://connect2id.com/learn/token-binding

Interesting. You already don’t have to worry about revoking JWTs if they’re sufficiently short lived. This gives you the exact level of protection as a short-lived mTLS cert, because if that gets stolen the attacker can continue to establish connections until it expires, unless as you say you revoke the certificate. So clearly I am missing something.

They key difference is with mTLS approach you'd have to steal the private key of the client certificate if you want to impersonate the client. In most secure deployments of mTLS and short lived certs, private key never escapes the TPM, Secure enclave or Yubikey, so it's extremely hard to mount an attack and impersonate a service.

With JWT (assuming it's not bound) you can steal JWT token and re-use it until it expires.

Right! Which is why we use (public) short-lived JWTs and (private) long-lived refresh tokens. What’s missing?

I still don't get the difference.

If someone steals your secrets you're screwed. No matter what kind of secrets that are. That's clear.

But if you keep your secrets in a HSM (TPM, SmartCard, …) and only use them to derive session keys directly on the secure device there is absolutely no difference which concrete tech you're using (given that secure cryptography is in place).

mTLS is a great approach, no question. But I just don't see how it's more secure than any other public key crypto.

mTLS tends to add infrastructure complexity, I much prefer it when you can "terminate" auth where you like in the ingress path for user-facing stuff rather than first-hop auth.

I wish there were "real" standards and a better ecosystem for request signing.

If I'm not mistaken, the difference is that in the case of JWT, your app manipulates the secret directly, so it must show up in clear form somewhere, from the app's perspective.

So, if the app host is compromised, the attacker shouldn't have too hard a time to extract the JWT and use it from somewhere else.

In contrast, with an HSM, the attacker would need to have the HSM sign any new connection attempt, which should be a bit more involved if it happens on a different machine.

Frankly, I think it will take years to replace API-keys (if it will ever happen). Developers are much better-off using CLI tools that prevent leaking secrets by blocking commits to git (e.g., https://github.com/Infisical/infisical or https://github.com/trufflesecurity/trufflehog)

I don't think those are mutually exclusive options :) Most developers, especially with lots of legacy apps are better off using a secrets manager. But there is no reason to not push the boundaries of security for new software and onboard passwordless and secretless options.

P.S.

I tried Infisical a couple of months ago. I think if I was Hashicorp Vault team's PM, I'd be worried. Your team has done such a great job at U.X. I was astonished to see an early startup with such a great integration catalog. I think you aced it - modern developers are desperate for out of the box integrations with 100+ services they have to use every day.

Wow! Thanks you Alex. This feedback means a lot coming from you! We're huge fans of Teleport, and learned a lot from you as a fellow YC company :)

No problem! Keep it up with out of the box integrations, focus on U.X. and developer experience and I think you will be on track to become as big or bigger than Hashicorp :)

How do you generate the short-lived certs?

Even if you try and use best practices, the whole ship is just LEAKY!

For example you store your secrets in the env. Well your program crashes and the log capture software dumps the entire env, or it’s included in some crash report.

Leaks from every corner.

We use two factor leaks.

Put an encryption key in the code and use it to decrypt the environment on use to recover the API key. You have to get both leaks to recover the key.

You have to combine that with some kind of assurances that the window that an key leak happens over is pretty finite, otherwise you can leak one key and not notice and you've lost your redundancy and you're no better off. And you may make incident response mistakes due to a false sense of security when you're made aware that the second key leak has happened.

So, you really need to be doing everything else correctly first and can't use that to fix sloppy key management and monitoring/prevention.

I’m the founder so naturally a bit biased, but with EnvKey[1] you can get pretty strong assurances by using IP restrictions. Even if an attacker gets hold of an ENVKEY token (used to fetch and decrypt an app’s secrets), if you lock it down to the IP range of your severs, they’ll only be able to use it if they can get inside your servers (at which point you are fully compromised regardless). Just getting the key by itself isn’t enough.

I think this is about as good as you can do, honestly, since even using something along the lines of short-lived certs, you still have secrets to protect, and you lose the ability to easily do immediate revocations.

1 - https://envkey.com

This seems implausibly high. Is it including stuff like putting password=replaceme in an example config file?

Well ever since github switched from password auth to token only, it's been impractical to do anything but copy paste it from somewhere and have it cache/store instead of typing it in from memory every time like one could do with passwords. Probably one of the worse security decisions I've ever seen, reducing github security to the same kind of level that those corpo directives do that need password switching every month, so it's just kept on a post-it on the machine lol.

No surprise that they get leaked all the time now, though at least one can cancel them at any point... and then you have to spend an hour or two replacing the cached version everywhere.

If the numbers look suspicious to you, I would suggest you check what is found more specifically about your company/organization.

GitGuardian can provide an automatic audit of your company-specific leaks we found on GitHub. Just ask: https://www.gitguardian.com/complimentary-audit-secrets-leak...

More details on how it works: https://blog.gitguardian.com/github-secrets-leak-free-audit/

Does this include deliberately sharing an API key, but in a 'not best practices' way?

ie. "Here, just run document.cookie='SID=EB73542386AF235' " Then you'll be logged in as an account that can do what you're trying to do.

No. This checks public GitHub commits that contain secret credentials; in 2022, over 33% were specific to known APIs, such as AWS access keys or MongoDB credentials, while the remaining 67% were deemed "generic." According to the report, generic ones include secrets such as "a company email and a password that would end up hard-coded in a file."

It's not scanning developer support tickets to see when devs hand over credentials to specific individuals.

Methodology is super important here though. I've used static analysis tools that identify every single bit of fake unit test data with a field called 'username', 'email' etc as leaked credentials.

Good point. I wish they included more details about how they performed their searches, but I understand not wanting to feed tips and tricks to the bad guys...

Yeah, for $curious_reasons we shipped our unit tests with our binaries and fake credentials would set off enterprise security scanners (although in our case it was a real enough looking PKI cert and not just data with username/password fields in it).

It looks like the best description of the methodology is here: https://docs.gitguardian.com/secrets-detection/detectors/gen...

It's pretty much checking if there's a variable called "_token" or "_secret" or similar which is then assigned to a string matching a regex looking for high entropy random-looking values.

It’s way higher than that if you mean just leaving it in an MR or even checking it in. Or even a relatively open internal file system. It’s not if, it’s about the plan when you do.

and that's why you use secret managers like infisical

We migrated a few month ago from doppler to infisical as the latter supports end-to-end encryption and happy so far!

EnvKey[1] is another option (I’m the founder).

We have similarly simple UX but are more robust on security. Browser-based end-to-end encryption as offered by Infisical is a bit of a fig leaf—it doesn’t protect against insider threats.

1 - https://envkey.com

I wouldn't undermine and characterize it as fig leaf personally. I think this would raise a natural question on my end which is how do you evaluate the security of platforms like 1Password, Bitwarden, Vault, all of which also offer browser-based alternatives?

Well, what’s the point of end-to-end encryption? It’s so that even if the service gets hacked or has an employee go rogue, sensitive data will still be safe, right?

That’s what the average person thinks they are getting when they sign up for a service that advertises end-to-end encryption, but when it’s browser-based, that simply isn’t true. If a browser-based service gets hacked, it’s trivial for an attacker to disable the encryption, steal private keys, or just steal secrets directly, and it can be done in a way that leaves no record and is very difficult for the end-user to detect.

So you have a security measure that is implemented in a way that fully subverts the stated purpose of said security measure. I think fig leaf is a fair characterization.

You’re right that Infisical is far from the only product doing it. Though it’s fairly common knowledge in the security and encryption community, most developers and most users of these products aren’t aware of this issue, so for now, there isn’t a lot of incentive to do it right.

First off, I want to point out that browser-based services and applications can be made secure; we should keep this one clear, especially for readers here, because otherwise it would invalidate most if not all services that exist accessible via browser. Take AWS which houses infrastructure for so many organizations; we access it via browser and this is totally fine.

The assumption you highlight is specifically "if a browser-based service gets hacked." Well yes, if any browser-based service, like AWS, got hacked then it'd be all game over. However, given that appropriate security measures are implemented/taken, we can in fact design web applications that are securely accessible via browser; this is the basis for how we access applications on the web securely, those that are not end-to-end encrypted (E2EE).

What I'm trying to say is that your assumption is based on a particular case that is "if a browser-based service gets hacked." I think, however, if this assumption occurred to many other non-E2EE services we use on a daily basis, we'd have a huge problem as well. Now, the reverse assumption that is the browser-based service is not hacked, coupled with an E2EE architecture, I believe it's possible to design a secure system here where even the server cannot read the values of secrets which is a point of E2EE.

Sure, my point wasn't that a browser-based product is automatically insecure, just that end-to-end encryption implemented in a browser has minimal security benefit. The whole point of E2EE is to avoid leaking data even if you do get hacked or have an employee go rogue despite your best efforts.

"even the server cannot read the values of secrets"

I have to disagree there. Your server can easily read the values of secrets--it just needs to include an extra snippet of JS in the response to a single request. You're asking users to trust that your server won't do this, but they have no practical way to verify it. That isn't the case for EnvKey, which is all I wanted to point out.

Please don't take it the wrong way--you have clearly built a product that has good UX and that people like and I congratulate you on that. Many users out there will probably prefer the tradeoffs you've made, while others will prefer EnvKey's. I think it's fair, in the spirt of friendly competition, to highlight where those differences are so that people can make up their own minds with an accurate understanding of the threat model of each approach.

Infrequent posting low karma users showing up on a random thread after nearly a year of inactivity to call out a relatively niche player which gets immediately replied to by the founder looks suspicious, I have to say. (EDIT: The founder has since deleted their comment, it had been quite downvoted before I replied)

I skimmed the report pdf and saw no mention of validating the data. So I assume pushing an example env file would be flagged as a leak? I understand that it's tricky to validate and even more so when having millions of data points but the method seems shaky. It's like all those automatic error analysers that repo authors tend to hate due to all false positives.

Definitely tricky to validate. Most tools out there offer multiple methods of excluding certain data-points whether it be by adding a comment by it to indicate intentionality, a file containing specs to exclude like values or matching patterns, or a UI that you can use "post-leak" to dismiss it as false positive.

I definitely resonate with your point though on the automatic errors due to false positives. There is a lot of room for improvement.

100% developers leaked JWT token .

Aren't JWT tokens supposed to be public?

Honest question, in terms of grammar, is JWT token similar to ATM machine? I always feel weird just saying “JWT” and almost always add the “token” part to the end for clarity, is that common?

> RAS syndrome (where "RAS" stands for "redundant acronym syndrome", making the phrase "RAS syndrome" homological) is the redundant use of one or more of the words that make up an acronym (or other initialism) in conjunction with the abbreviated form.

> Many style guides advise against usage of these redundant acronyms in formal contexts, but they are widely used in colloquial speech.

https://en.wikipedia.org/wiki/RAS_syndrome

I try hard to just say "the JWT". But I also try to only say "ATM"

Bonus points if you pronounce it like a Greek "ω" (joat) or a Welsh "w" (joot)!

That's omega?

you're telling me the w doesn't stand for rapture?

Depends. Is your name Wodger or Bwian? Do you have friends in Wome?

it’s a joke about the new testament.

So is this, indirectly: https://youtu.be/7Lc86JUAwwg. Though sadly it didn't include the Book of Wevelations.

no, just a play on omega, but thanks for the nostalgia and the laugh :)

I go the other direction and like to say JWT Web Token.

"Personal PIN Identification Number" is my favourite example.

No? They are literally your authentication. They are time limited, but still pretty private.

Done well, they have an audience set for what they authenticate. Which can limit the exposure back to the issuer. But you are supposed to trust that holding the token means you know who the user is.

Right?

Depends on the context. JWTs are very often used as a type of session token. So if your token is public, it might allow someone to act on your behalf for whatever service you're using.

So while users are allowed to read the token and its contents, you don't want user-identifying tokens to fall in the hands of someone else.

You might be thinking of a public key.

Reply on news.ycombinator.com