Stealing unencrypted SSH-agent keys from memory | Hacker News Reader

Stealing unencrypted SSH-agent keys from memory

netspi.com

81 pointslelf38 comments

I've upvoted this because I want to see what really smart people have to say about it. That said, I've always been of mind that if someone gains unauthorized root access to my box, then nothing on it can ever be trusted again. This includes things in memory. In other words, this feels obvious to me, once pointed out.

Is there something novel about this that I'm missing?

No. If you have the ability to run code as root it's game over. Even if the agent re-encrypted the key in memory with its own temporary key it's just a layer of obfuscation that can easily be overcome.

I agree with you, but in case you missed the author's rationale:

"However, this causes the attacker to have to wait for the target to type in their passphrase. This might be hours, days, or weeks, depending on how often the target logs out. This is why obtaining the SSH key from memory is vital to pivoting to other machines in a speedy fashion."

I'm not sure how you would solve this problem while providing the functionality ssh-agent provides, aside from perhaps a HSM or something.

Anyone keeping SSH keys in an ssh-agent process for weeks on end is doing it wrong. You can use the -t option to ssh-agent to specify a maximum lifetime for identities added to the agent. I would set it to less than a day.

if you unlock it as soon as it expires im not sure what that gives you. all the attacker needs anyway in that scheme is 24h, its not a big difference.

Make sure nothing can run as root? Perhaps by using a VM system. You could run ssh-agent on another machine and make key requests over an internal network.

Also you can get a cheap "HSM" by using a smartcard.

The problem of a root user inspecting the memory of arbitrary other users is solved with MAC like SELinux. https://security.stackexchange.com/questions/7801/keeping-se...

But if you can exploit a specific user's program, you can (usually) inspect the memory of other programs managed by that user. You can actually create security policies so secure that a user who exploits the sshd program can't read from the same user's ssh-agent process memory, but that's not practical for most people.

It's not that novel. If an attacker gets root it's game over.

That said, the article is a good reminder of the risk of keeping decrypted keys in a long-running ssh-agent process.

nothing's novel - but consider this for example:

- if its a server and you copy your private key onto it its likely to be compromised one day and the key reused even thus it was encrypted.. since its decrypted in memory when used. plus keystrokes could be captured. if you use this key elsewhere, all other servers are also compromised now..

- you can keep the key in a hardware token, then even if a laptop is remotely compromised your key - and thus servers you're not currently accessing - are safe. its also a little harder to hijack a connection or silently ask the key (generally its not silent with a token)

I wish SSH had a narrow kind of delegation capability... E.g. if instead of ssh-agent having access to the key, it used the key to sign a tuple like {time-limit, [host-ip], ephemeral_pubkey}, then forgot the private key, and used the ephemeral key to actually authenticate to hosts— but they'd only accept it from the listed host ip set and during the specified time limit.

Obviously if the attacker has access long enough they'll capture the passphrase and private key eventually, but it would be nice to narrow the window.

Right now the best you can probably do is use a gpg smartcard with the ssh-agent/gpg-agent shim stuff, (random tutorial here: https://github.com/herlo/ssh-gpg-smartcard-config/blob/maste...)

Your scheme could probably be implemented using certificate based SSH: Your users log onto a CA which gives out user-certificates with a specified lifetime, they copy that to their desktop-machine(s), or maybe add it to a ssh-agent like intermediate key-storage and can do their days work.

Tomorrow, they'll have to get new certs (automatically, after authenticating to the CA using appropriate means).

Here's a tutorial showing cert-based ssh auth: http://neocri.me/documentation/using-ssh-certificate-authent...

Obviously with secure key storage (smart-card) prohibiting accidental loss of the private key, and a working cert-revocation infrastructure it's not necessary in the first place.

That's essentially Mozilla Persona, but for SSH instead of the web. The people at Mozilla are in fact already discussing such possibilities: https://hacks.mozilla.org/2013/04/mozilla-persona-for-the-no...

>I wish SSH had a narrow kind of delegation capability... E.g. if instead of ssh-agent having access to the key, it used the key to sign a tuple like {time-limit, [host-ip], ephemeral_pubkey}, then forgot the private key, and used the ephemeral key to actually authenticate to hosts— but they'd only accept it from the listed host ip set and during the specified time limit.

So Kerberos? You can do that with Kerberos and the GSS-API in OpenSSH already.

So, as someone needing to run unattended scripts across a wide variety of hosts, what's the best practice for managing SSH keys? I've been using ssh-agent. Is there a better way?

Make sure that the SSH keys are tightly restricted in the authorized_keys file so they can only run the commands that need to be run unattended. If possible, also restrict them by source IP address. Here's an example authorized_keys file that I have in production for doing unattended backups:

  command="/usr/local/backups/backup_server /etc/snapshot_backup_list",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,from="10.70.0.0/16" ssh-rsa AAAAB...

I wouldn't worry unduly about protecting the keys themselves - since they need to be accessible for unattended operation, there's not much you can do to prevent them from being accessible to an intruder.

This is hard, since there is such a large number of them. It would be very painful to maintain this list. For a while I had the keys expire in ssh-agent, but that didn't work because you can't extend the expiration before you run a set of unattended commands. That is, if it's 11.5 hours into a 12 hour expiration, and I need to run commands that will span the expiration, then I'm screwed, since 1) I can't query when the keys will expire, and 2) I can't extend it another 12 hours (from now, not from the expiration).

I feel like ssh-agent is essential to my life but it's badly designed and I have this nagging feeling it's insecure (and articles like this feed this fear).

That sounds like a bit of a quandary. One thing I'm not clear on - if these run unattended, how do you enter the passphrase?

ssh-agent isn't insecure, it's just not magic. If an attacker gets root access to a box, they can examine the system's memory, and the memory of ssh-agent necessarily contains the unencrypted private keys.

Edit: if you're really concerned about private keys being exfiltrated, you can always use a smartcard (the OpenPGP card[1] in a Gemalto USB Shell Token[2] works well with SSH). But if the smartcard is online all the time, then an intruder can always simply use the smartcard to SSH wherever they want, even if they can't actually get the private key itself.

[1] http://shop.kernelconcepts.de/product_info.php?products_id=4...

[2] http://shop.kernelconcepts.de/product_info.php?products_id=1...

I think the point is that they enter the passphrase once to load the keys into ssh-agent, and then the unattended process can use them forever.

If that's the case, then I would start a dedicated ssh-agent process just for the run, add the identities (without expiration), and later kill the agent when the unattended run is over.

That's fine for that kind of scenario, but often "unattended run" means something spawned by cron, like a daily backup, or a CI system that spawns build jobs on slave machines. There's not a lot of options there.

This is the case.

> I feel like ssh-agent is essential to my life but it's badly designed and I have this nagging feeling it's insecure (and articles like this feed this fear).

If the attacker has root-access, like in this article, they could also recover your decrypted private keys from the memory used in active connections. (SSH and SSL both)

I would hope that ssh zeros the memory used for the private key as soon as it's done with it (which would be after the authentication step in the handshake).

But then it would have to either keep the passphrase in memory (just as bad) or ask for the passphrase each time you reconnect later (defeats the purpose of ssh-agent).

Edit: Nevermind, you're talking about ssh, not ssh-agent..

You're right, I forgot that it would switch to asymmetric crypto after the handshake. My bad.

Still, with root it would be trivial to attach a debugger to the daemon, et cetera.

Not really a solvable problem. You want your machine to have the keys to run scripts but don't want your compromised machine to have the keys to run scripts. The best you can do is restrict what the keys can do or use authentication that doesn't work unattended.

If you're just doing batch jobs, then you could have the script remove keys from ssh-agent when it's done. At a certain point you have to presume the integrity of your machine. Otherwise your password can just be keylogged as you're unlocking key.

The technique of using gdb to attach to ssh is pretty old.

Here's an example from 2005, which was presented (iirc) at Defcon as well: http://www.blackhat.com/presentations/bh-usa-05/bh-us-05-boi...

In that case, SSH-Jack would just piggyback on existing (user-level) ssh connections, which is also pretty serious, though that's not as exciting as stealing keys.

personally i find it more exciting than steal the key because harder :)

then again this as been done since the beginning of times sadly, or excitingly, i dont know :)

any tool that promotes awareness of this inherent issue is good anyway IMO

Yep. The general technique of scraping memory for private keys and certificates a pretty well-known and established. Particularly in forensics and malware circles.

I don't point it out to engage in the "who was first" thing. But to point out that this is very much an applied attack in the real world. Real attackers (includes "forensics analysts", incase you don't consider them "attackers" too) have been using this technique in malware as well as countermeasures/investigations for quite a while, now.

See: http://www.trapkit.de/research/sslkeyfinder/

https://github.com/emonti/yara-ruby/blob/master/samples/sslk...

http://volatility-labs.blogspot.com/2013/05/movp-ii-21-rsa-p...

EDIT: actually a much earlier discussion is from '98 by none other than Shamir

https://www.cs.jhu.edu/~astubble/600.412/s-c-papers/keys2.pd... [PDF]

This is exactly why SSH has agent forwarding.

Do not copy your private key to other systems.

ssh -A is your friend.

Well, it prevents your private key from being compromised but an attacker with root on the intermediate box can still authenticate to other servers using your SSH agent socket.

If you really want to do secure SSH hopping, use the ProxyCommand directive with `netcat` in your SSH config.[1]

[1]: http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Ju...

Nice. Thanks for sharing

So, is there a big advantage to running ssh-agent vs unencrypted 0400 keys in your ~/.ssh directory?

Yes. ssh-agent prevents non-root users, including the user running ssh-agent[1] from being able to extract the private key. That's much more secure than storing unencrypted keys in ~/.ssh.

[1] ssh-agent uses some tricks, like making itself setgid, and/or using prctl(PR_SET_DUMPABLE, 0) on Linux to make its memory undumpable.

And on OSX, it integrates with the keychain as well!

You're better off not storing your passphrase in the keychain and just typing it in that one first time, though. But it's still marginally better than an unencrypted private key for automation if you need a 100% unattended setup.

"In the keychain" doesn't necessarily mean in the login keychain. You can create as many user keychains as you want, with different passwords. Also, add the keychain app to your menubar (via preferences) to get a quick way to manually lock one or all keychains (and alternatively, the whole computer).

That won't prevent your key from being stolen if you leave it on a VM controlled by someone else.

Your key belongs on your personal device, and agent forwarding enables you to behave as if your key is everywhere you need it to be.

Reply on news.ycombinator.com