You don't want long-lived keys | Hacker News Reader

You don't want long-lived keys

argemma.com

79 pointskkl62 comments

Okay but now how do you recommend I hook up my Sentry instance to create tickets in Jira, now that Jira has deprecated long-lived keys and I have to refresh my token every 6 weeks or whatever. It needs long-lived access. Whether that comes in the form of a OAuth refresh token or a key is not particularly interesting or important, IMO.

You do what you can. Eliminating long-lived keys isn't always possible; you set up rotation instead.

Does having to refresh the key every 6 weeks instead of every year or whatever actually make a meaningful difference security-wise?

Yes? That's a huge difference.

At the minimum you’ll remember how to do it if you have to do it every six weeks.

If the key becomes compromised, rotating the key sooner means you potentially limit the damage from unauthorised access.

I disagree, I think increasing manual toil (having to log into Sentry every 6 months to put in a new Jira token) increases fatigue substantially for, in this case, next-to-no security benefit (Sentry never actually has any less access to Jira than it does in the long-lived token case, and any attacker who happens to compromise them is going to be gone well before six months is up anyway).

Instead, the right approach in this case is to worry less about the length of the token and more about making sure the token is properly scoped. If Sentry is only used for creating issues, then it should have write-only access, maybe with optional limited access to the tickets it creates to fetch status updates. That would make it significantly less valuable to attackers, without increasing manual toil at all, but I don't know any SaaS provider (except fly, of course) that supports such fine-grained tokens as this. Moving from a 10 year token to a 6 month token doesn't really move the needle for most services.

This sounds more like a reason to automate token management than an argument for long lived tokens.

But then you just move the security issue elsewhere with more to secure. Now we have to think about securing the automation system, too.

This is the same argument I routinely have with client id/secret and username/password for SMTP. We're not really solving any major problem here, we're just pretending it's more secure because we're calling it a secret instead of a password.

Secrets tend to be randomly-generated tokens, chosen by the server, whereas passwords tend to be chosen by humans, easier to guess, and reused across different services and vendors.

How does this apply to ssh public keys?

> Long-lived production SSH keys may be copied around, hardcoded into configuration files, and potentially forgotten about until there is an incident. If you replace long-lived SSH keys with a pattern like EC2 instance connect, SSH keys become temporary credentials that require a recent authentication and authorization check.

It’s like 12 lines of terraform to fully automate this, inside your existing IaC infrastructure. It’s not complex.

Seems like you work in an organization with one developer. It's never just a "small configuration change" when you need to change the workflow and habits of your entire company.

I work for a gigantic tech company, and it’s still just a small configuration change.

What’s your excuse?

Why? If Sentry gets compromised, it's the exact same outcome—your Jira tickets get mined for production credentials and downloaded for random. What does automated token management save here?

How long the Jira access lasts depends on you / Sentry detecting and solving the initial intrusion. It doesn't matter how long the Jira token itself lasts if the attackers have access to the database in which its stored or log files in which its been dumped or something like that.

This assumes that the intrusion is persistent until Sentry catches it, that Sentry notifies me, that I successfully track all the places I have long-lived tokens that need to be rotated, etc.

OIDC with JWT doesnt need any long lived tokens. For example, I can safely grant gitlab the ability to push a container to ECR just using a short-lived token that gitlab itself issues. So the answer might be to ask your sentry/jira support rep to fast track supporting OIDC JWTs.

- https://docs.gitlab.com/ci/secrets/id_token_authentication/#... - https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_pr...

My favorite are the systems where you can only issue one token, so that you can't do a zero downtime rotation by creating new one, making it active in your system, and only then removing the old one.

In some cases this makes rotation a big event to be avoided because costs are higher than gains.

I am still surprised that Keycloak makes this so hard. They finally added support for n=2 but it’s still walled off behind a “this is experimental, use at your own risk” warning, and it’s something that literally every OIDC client needs to do if you have any kind of compliance requirements.

> If you assume that someone is constantly trying to guess a key or password, the likelihood that they guess correctly grows over time.

If they can brute force the password or key, the rotation will, at best, force them to do it multiple times. You'll see more improvement from just adding another couple of characters to the length.

Fair enough, but that doesn't protect you in case of a leak. If you're going to solve for the leak anyway, is it worth it to solve for brute force in isolation? You can always add another couple of characters. At which point do you stop?

On the contrary. We want long-lived keys. As long as they are not symmetric!

My private SSH key is rooted in hardware and can't even be extracted. This is awesome, I don't have to worry about it getting compromised.

The same should apply to all other keys, including the dreaded "bearer tokens".

I’m sorry to be pedantic, that’s not exactly true. I agree in the sense that extracting hw based keys is next to impossible, but if your machine is compromised, there isn’t much stopping malware from using your hw based key (assuming 1. Left plugged in, 2. Unlocked with either ssh-agent or gpg-agent, and 3. You don’t have touch to auth turned on). Reduced risk? Absolutely. No risk? Absolutely not.

Sure. They can use my key while my machine is compromised, but even then I won't _need_ to rotate it after the compromise is cleared.

It still would be a good idea just to make sure that it's easier to analyze logs, but it's not strictly needed.

And if you want to be even more pedantic, shell access with a touch based key just means the attacker has to wait for you to auth, which makes touch based systems largely a waste of effort on the defenders part.

> shell access with a touch based key just means the attacker has to wait for you to auth

And if you want to be EVEN more pedantic, on most touch-based keys, you have to touch within 10–15 seconds otherwise it times out.

So it is not a waste of effort at all. First the need to touch at all eliminates a large chunk of attacks. Second the need to touch within 10–15 seconds eliminates a whole bunch more.

There would have to be some heavy-duty alignment of ducks going on to get past a touch requirement.

Even more if the target has touch AND PIN enabled.

The touch based key I use only responds once per touch. If someone compromises the machine it's plugged into, the action I expected to complete won't complete. This means the compromise becomes immediately visible.

Never apologize for pedantry here

> there isn’t much stopping malware from using your hw based key

Except the three pretty major things that do stop malware that you mentioned ;)

Perhaps especially "3. You don’t have touch to auth turned on".

Part of the threat model for an Engineering team is that people come and go. They move teams which have different levels of access. They leave the organization, in most cases, on good terms. I want to set up infrastructure where I don't need to remember that your SSH pubkey is baked into production configuration after you leave the company.

There are several options for setting up per-connection keys that are dispensed to users through the company SSO. That setup means you don't need to maintain separate infrastructure for (de-)provisioning SSH keys.

This is completely solved by SSH certificates. You still have the same private key in the hardware, but instead of using the public key directly, you issue temporary (~1 hour) SSH key certificates. I even automated it using an SSH proxy.

The target machines then just need to put the CA cert in the authorized_keys files.

> The target machines then just need to put the CA cert in the authorized_keys files.

The word "just" is doing a lot of work there. You update authorized_keys every hour for your entire fleet?

Sounds like a job for dnssec and sshfp records

Ahh, now you have three problems…hrm

No, the ssh CA model works like this: servers trust one CA, and the CA signs user keys. No more distributing individual public keys to every machine.

It is the user machine that needs new certificate signed by the CA once the short-lived one expires.

Understood. Not a bad idea.

SSH CAs are not incompatible with my argument! An SSH CA can sign short-lived certificates for just-in-time generated keys.

shameless plug: if your are looking for software that does basically this, you can try my open source project: https://github.com/azophy/sshifu

Its still early, but my aim for this project is to be the simplest/easiest "SSH login via SSO" available on the market. open for feedback & suggestion

Agreed! Been working on infra for an early-stage company recently and it's been awesome using OIDC and IRSA (or WIF if you're on google) for as many things as possible. Basically, there are no permanent keys for anything.

Slightly annoying to have to wrap some clis in scripts that generate the short-lived token, but it feels really magical to have services securely calling each other without any explicit keys or password to even store in our vault.

Lots of cool benefits --- for instance, we ran the compromised Trivy github action a few weeks ago, but our Github Actions had 0 keys for it to leak! Also really great that I don't have to worry about rotating shared credentials on short notice if an engineer on my team decides to leave the company.

Where possible I prefer to implement signed policy objects. Then I can constrain access based on source IP and other request parameters. You can also easily implement an expiration date if you feel any particular application requires it, but some simple constraints may be useful enough that you might skip this in the majority of server to server applications.

This not only provides security but provides some resistance to bugs in your code which either call services incorrectly or call the incorrect methods of a service. I've avoided accidental data deletions and other painful events because my policy document did not allow this action. It turns these bugs into failures at the security perimeter.

I've used this concept in a few user applications as well. Typically those documents will always have expiration dates and I'll provide a "license" API which allows a single authenticated client request to retrieve an appropriate policy document. This is particularly nice when you want to implement a service across different providers or want to avoid downstream authentication overhead.

Even still, I prefer the simplicity of API keys. The mental overhead is low, and having to explain the concept to customers is zero. Rotating keys is not great but the tedium of it is preferred over the labyrinthian shenanigans of getting whatever the hot new security style of the week setup is.

A case where I am fine with long lived keys is a public static blog that has no sensitive data and does not allow user contribution of any kind. A site so static and public I can check 100% of the files and web server configuration into a public git repo minus the private key. That's just my take and my personal preference of course.

Honestly I would use a self signed certificate if there was a way to tell the browser it is a static blog and not to add friction to the user experience. It's not like LE is adding identity verification to the process any more than a self signed certificate would. It would be nifty if there was some naming standard that could tell the browser a blog is absolutely static and free of anything sensitive like staticblog.domain.tld so that it could dial down the you are about to be pwned! which I am sure some would abuse but that's fine, let them.

After the Vercel hosting compromise and having to rotate a ton of keys recently, we are definitely implementing automated rotation of short lived keys. That was super painful.

But how do you do that without also having a long-lived key or access token to those services?

The long-lived credentials life inside a stripped down machine. Cron/lego/Ansible handles the renewal. The machines on the edge can't renew their keys themselves.

Oh, this makes sense, so instead of "the app is rotating its keys" is more like "the keys in our app are being rotated by an external service".

I find it interesting how this all comes down to what do you trust. Like.. why not <1 minute keys? Or 1-request?

We do a lot of request signing (think AWS sigv4a) which practically speaking amounts to 1-request.

> why not ... 1-request

You kid, but: https://en.wikipedia.org/wiki/Hash-based_cryptography

also: https://www.imperialviolet.org/2013/07/18/hashsig.html (2013).

What about dynamic credentials. Why can't we deploy HSM(hardware security module), they are so much more affordable now. We then deploy fido2 keys, have our long lived keys in there and have HSM serve as dynamic credentials server.

one thing that always felt off to me is how much focus goes into how access is granted and rotated, and much less into what actually happens after that access is used

even with short-lived or well-scoped credentials, the behavior behind them can still be pretty opaque depending on the system

so you reduce risk on access, but what happens after can still be hard to reason about or observe

I think the take on key lifetime is premature which taking into consideration

1. How key is used

2. Whats the threat vector

3. Cost of key rotation

4. Cost of key verification

At the end of the day its a trade off, the business use case, your expertise and the risk have to be evaluated together

Something I don't understand is the absolute phobia of service accounts. There are things that need to happen regardless of who is doing it. Emails need to get sent every day with reports, for example.

Forcing these workflows into the nonsense security theater of "we can't have service accounts" is stupid and unproductive. So every time we fire or lay off the person whose name is on the automation, we need to rotate the keys? What is the benefit here?

If you are screaming "managed identity" here, I have a bridge to sell you because clearly even Microsoft has not been able to figure out or implement managed identities for internal workloads... Well not as of 2022, at least.

Totally, but my service accounts own the api keys. But keys are still annoying to rotate. You know what’s not annoying to rotate? Short-lived tokens with very limited scope that get assigned more on demand

Service accounts are great! I just wish instead of having a password which gets shared around via 1password, there were a clear permission list ("this is a service account.. "real" users X, Y, X can login as it")

Seems like it's just Microsoft that cannot figure it out. AWS had roles forever, fully supported from web console or CLI. But when I request Azure service account, I am handed username and password.

> "we can't have service accounts"

To be clear: This is not my position! I advocate for service accounts in my post:

> It is much harder to reason about, say, the security of an arbitrary Engineer's laptop than it is an EC2 instance that exists exclusively to tell KMS to sign something.

> So every time we fire or lay off the person whose name is on the automation, we need to rotate the keys?

If a person previously had access to the key and knowledge of the key gives you control over that automated workflow, is that key (and by extension that workflow) still worth trusting?

You dont usually want keys at all. At least in the sense of copy this key from system A and paste it in this other place system B. Usually CI. You want some continual method of authentication and authorization.

Some magnificent systems have APP_KEY/APP_SECRET that is also used for cookie and database encryption. A frequent rotation of this is... inadequate... in systems with high traffic, to say the least, and hence I am sorry, but I do not believe it's the "usual" desire. As always, it depends on the context and transaction scope.

  Related:
  - 1. https://symfony.com/doc/current/reference/configuration/framework.html#configuration-framework-secret
  - 2. https://laravel.com/docs/13.x/encryption#gracefully-rotating-encryption-keys

The fixed position background made it look like I had dust on my phone screen

It didn't for me, and I got the starry space feel, but I noticed the repeating patterns.

Perhaps some movement is needed? I do recall some relatively similar cases saved, if interested:

  1. Moving forward in space (JavaScript/JS): https://codepen.io/the_artwork/pen/zYEdxyo
  2. Rotating in space (JS): https://codepen.io/the_artwork/pen/NWMRYJP
  3. Rotating in space (CSS+JS): https://codepen.io/the_artwork/pen/PoeNyyy

I'm still having problems trusting my compiler.

Not a single mention of SPIRE thus far, sus

Reply on news.ycombinator.com