SSH Agent Explained | Hacker News Reader

SSH Agent Explained

smallstep.com

393 pointssourishkrout48 comments

Thanks for this. Wanted to put in a pitch for Dima Kogan's more-secure way of doing ssh-agent forwarding: https://github.com/StanfordSNR/guardian-agent

It works with SSH and Mosh. The basic idea is that before agreeing to a request, the principal or their agent should know (a) what machine is asking, (b) what remote machine they want to connect to, and (c) what command line they want to run on the principal's behalf. And the principal's authorization should then be limited to that context.

The ssh-agent protocol doesn't give the agent any of that information; it's really just intended for local SSH-like processes to ask "please sign this blob so I can connect somewhere" without them having to see the plaintext private key. Forwarding that to untrusted remote machines isn't ideal.

It turns out an agent can get access to this information and limit the scope of authorization in a mostly[1] backwards-compatible way, which is how guardian-agent works, but imo it would be preferable if a future version of the SSH protocol were designed more expressly for secure agent forwarding.

[1] For (c), the remote server has to be OpenSSH because guardian-agent relies on a nonstandard extension.

YES. It would be so easy for OpenSSH to fix agent forwarding. Just need to limit authority and/or ask for consent for a particular action.

You can configure ssh-agent to ask for confirmation if you set the `-c` flag in ssh-add or by setting `AddKeysToAgent` to `confirm` in your ssh config [1].

Once set, authentication will require confirmation via a GUI dialog provided by the ssh-askpass command. However, it does not mention the command or process requesting for authentication.

It works great on Linux, but I couldn't get it to work on macOS with the system keychain.

[1] https://man.openbsd.org/ssh_config.5#AddKeysToAgent

Recent macOS versions don't have `ssh-askpass`, and it's weirdly hard to add one. Since agent confirmation depends on askpass, I don't think there's an easy way to get this work on macOS.

Aside from the missing context you mentioned, the other bigger problem with this approach is that agent confirmation is all-or-nothing: it turns on confirmation for local SSH connections in addition to forwarded connections. If you're using SSH a lot, having to confirm every connection is very annoying.

> The basic idea is that before agreeing to a request, the principal or their agent should know (a) what machine is asking, (b) what remote machine they want to connect to, and (c) what command line they want to run on the principal's behalf. And the principal's authorization should then be limited to that context.

Holy snap this is exactly the reason that I've gone to great lengths to disable SSH Agents and askpass. Asking for a password without any context whatsoever of exactly which process is wanting it is a nightmare

At first glance guardian-agent seems to be focused on a happy path where the remote systems are honest. Clearly agent forwarding attacks would mostly involve dishonest (or at least corrupt) remote systems and if guardian-agent specifically defeats that rather than just punting I don't see how in this summary.

For guardian-agent to live up to its security claims, it needs to correctly deny requests on the "unhappy path." So yeah, that's a primary goal.

As far as we know it lives up to that. Requests are locked to (a) [identity of requesting machine] because their origin is tagged by the local (trusted) code, which got to see the requesting machine's public key when it connected. They're locked to (b) [remote machine] by having the local code see the remote machine's public key. And they're locked to (c) [command] by having the local code be the one to send the command and then "no-more-sessions" and get confirmation before handing over control to the requesting machine.

One thing I like to do with the SSH Agent is also forward my X.509 certificates by adding new opcodes to the SSH Agent protocol, then you can do stuff like PKCS#11 on the remote side with your local smartcard. [0]

This gives you, among other things, passwordless but authenticated "sudo" capability (which is actually required by the DOD, though nobody does it).

[0] http://cackey.rkeene.org/fossil/artifact/0d0e90bbfdee672c?ln...

Forgot to include a link the PKCS#11 module which talks to the SSH Agent: [0]

[0] https://chiselapp.com/user/rkeene/repository/ssh-agent-pkcs1...

Are you aware of any hardware tokens that support PKCS#11 and can be implemented affordably at small scale? The only ones I've found are Yubikey, but Yubikey's other protocols (OTP, U2F) are a lot simpler to implement, so I've never tried.

At one point I bought a few literal smart cards and a USB reader, but not being an enterprise customer of any of the vendors, couldn't find the necessary drivers / passwords to get them working with PKCS#11.

It's not 'simple', but it's definitely doable with Yubikey: https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PK...

I've used it on macOS to use a PIV cert on a yubikey to ssh to remote linux servers. I have NOT used it to ssh-agent chain through, jump-box style though.

If you're already bound to Yubikey, though, any advantage to PKCS#11 over OTP?

Theoretically it's an open standard, but there seems to be only one real vendor.

PKCS#11 is extremely versatile, with some configuration you can use Yubikey PIV X.509 keys to authenticate SSH sessions by passing the necessary pkcs11 lib to ssh with the -I flag. PKCS#11 isolates the key signing from the PC by doing any private key computation operations on the secure element, this prevents leakage of the private key by traditional means.

Browser TLS client authentication is also possible with some minimal initial setup of your browser of choice. You can use openssl with a pkcs11 engine for any X.509 operations you might think of (e.g. PKI, S/MIME email signing, etc.)—technically you could use the PIV applet to store web server TLS keys and have Nginx use those through OpenSSL via the pkcs11 engine, though it would probably slow down handshaking as the throughput and signing speed of a Yubikey is orders of magnitude slower than any modern general purpose CPU.

Many people don't know that Yubikeys also run a OpenPGP card applet for storing up to 3 key pairs. With the ssh option of gpg-agent it can also be used for SSH authentication just like ssh-agent.

I use this in combination with a Yubikey configuration setting requiring all GPG sign operations to be confirmed within 10 seconds by pressing the capacitive touch button on my YK4 nano (LED blinks during this time to prompt). This way I can safely enable agent forwarding globally as all SSH session authentication requires physical interaction to confirm (provided you don't have any file-based keys in the same agent which are still vulnerable to usage without your knowledge via this functionality).

I use the gpg-agent option as well, but hate it (GPG makes me feel icky). That is not, as I understand it, using an X.509 key for authentication though. Which is what I'd like (ssh-agent forward an X.509 based auth mechanism - like remote PKCS#11 over that agent connection, so the key stays on my yubi, and I only need CA certs on the remote hosts.

This is exactly what I do -- I posted some links to both the SSH Agent (well, the ChromeOS version -- I've got C versions as well) and the PKCS#11 module which talks to the SSH Agent. So on a remote system I can do things like sign files using PKCS#11 and passwordless sudo using pam_pkcs11.

There are MANY vendors that supply hardware security modules that make their information available via PKCS#11 modules. From smartcards to TPMs to dedicated large HSM devices.

There are many advantages of something like PKCS#11 over One-Time-Passwords (TOTP/HOTP):

1. Since you have an X.509v3 certificate, you have an identity not just an authenticator -- so when you are using this certificate to login to systems, you don't also have to supply an identification (though most systems let you optionally supply one, e.g., if you can login with multiple usernames and the same credential).

2. PKIX covers a lot more use cases than OTP, things like Kerberos (PKINIT), TLS (TLS Client Certificates), SSH (PKIXSSH et al), digital email signing (S/MIME), code signing, commit signing, and more

3. The device is basically a second computer, so it can run software policies on it to do various things like if you authenticate incorrectly a few times in a row it can zero itself out, with OTP systems that has to be coordinated by the various systems validating the OTP token.

It's so awesome, and the documentation and tooling suck so hard. Such a frustrating ecosystem...

The only real low point in the documentation I've found was around MIT Kerberos.

I implemented a PKCS#11 module from the documentation for NIST SP 800-73 and RSA PKCS#11 in about a week.

What things do you think could be better documented ?

If you're going the Smart card route you can get the PIVKey C910 on Amazon for $15/ea. in quantities of 1. You can also buy them for about $0.10/ea on AliExpress, and then load CoolKey or some other applet.

>Platform Support

>Windows In-Box PIV Driver (Read Only) and PIVkey Windows Minidriver (Read/Write): Windows Vista, 7, 8, Server 2003, Server 2008, Server 2012.

>Mac OSX, Linux and Android (Read Only - Middleware Required)

The only open source middleware I know of for OSX is OpenSC, and it doesn't list C910 as a supported card.

https://github.com/OpenSC/OpenSC/wiki/Supported-hardware-(sm...

The PIVKey C910 complies with NIST SP-800-73 (PIV) -- same as YubiKey PIV mode, so OpenSC supports it. Also, the middleware I wrote (CACKey) is open source and supports it and works on macOS as a tokend driver.

I forgot to mention that also, of course, CoolKey is open source and the CoolKey PKCS#11 module supports the CoolKey applet (obviously) and makes some attempts to support the US DOD CAC and the US Government PIV -- though I don't know how successful they are. Bugs and feature limitations in CoolKey led me to write CACKey.

Coolkey is trying to link OSX 10.5 libraries and cross-compile for PowerPC. Seems like abandonware. Are you using it successfully in 2020?

I mostly use CACKey, since I wrote it to replace CoolKey for my needs.

I regularly ssh into boxes with varying IPs. Something like "ssh -i <key> <user>@<ip>". Every time I want to scp a file, I quit the ssh session, press UP, modify the ssh command into an scp one, execute then restore the ssh session. It's slow and annoying.

Does anyone know of a way to use scp without hassle, once ssh session is established?

Use session sharing (ControlMaster setting). Then, don't quit your session - in another terminal on the same computer, run your scp command, and it will use the existing session (and its authentication), without need for -i.

Alternatively, ssh-add is your friend.

(But for security, make sure you do not forward ssh-agent unless you understand the risks)

You can setup an `~/.ssh/config` file to make it easier:

  Host *
   ControlMaster auto
   ControlPath ~/.ssh/ssh_mux_%h_%p_%r
  Host host1
   HostName 1.2.3.4
   LocalForward 45432 db.internal.net:5432
   User user1
  Host host2
   HostName host2.whatever.com
   LocalForward 46432 db2.internal.net:5432
   User user2

This way you can now scp/ssh to user1@1.2.3.4 with just this:

  scp file host1:
  ssh host1

It also setups up port forwarding. In that case I can connect a postgresql client to localhost 45432 and it'll forward (assuming the ssh host has network access to it) the tcp connection to the host db.internal.net . I do this to use GUI SQL clients to our back-end DB.

The Control options setup ssh multiplexing. If you ssh into a host, subsequent connections piggy-back off of the initial connection. This is useful if you have 2fa and don't want to do 2 factor for every connection.

There's much more you can do: https://linux.die.net/man/5/ssh_config

Something that's skimmed over in the article but not addressed is: if the key pair isn't used for encryption, then how are session keys protected?

The answer is: using the server's public key which is transmitted to client when establishing the connection.

But then it's trivial to perform a person-in-the-middle attack and both observe and manipulate the plain text data by sending the client the attacker's public key.

That's why it's crucial to retrieve host keys via secure channels and explicitly whitelist them on clients.

"Protected" is an odd word to choose here because session keys are agreed long before we know who we're talking to.

The approach of SSH (like modern TLS) is to create a secure channel between two participants and only then authenticate one or both participants by binding credentials to this secure channel.

The article gets that upside down, which is understandable because most people seem to imagine that it'd be essential to figure out who you're talking to first and only then encrypt things, but actually the opposite is better.

If you do Trust On First Use as many SSH users do, then you're correct that bad guys can interpose on that first connection if they happen to get lucky - but that's because they can authenticate as the "correct" server by presenting their own public key since you have no idea what the correct one looks like.

>"The answer is: using the server's public key which is transmitted to client when establishing the connection."

This was true in SSH v1 which is ancient but in modern times v2 uses DH and the the server's pub key is only used to sign the DH parameters.

You can use: "monster in the middle", mitm, if you want to keep it gender neutral.

The author states:

>"Later in the handshake process, a set of new, ephemeral and symmetric keys are generated and used to encrypt the SSH session traffic."

This isn't correct. After the TCP handshake, both systems agree on a session key by using the Diffie-Hellman key exchange. It's only after establishing a session key that user authentication takes place. There's a re-keying event but that's not part of the initial handshake and only happens later when some threshold(time or blocks) has been reached.

The problem is that this breaks with use of tmux or similar tools since the scope of the SSH session is often smaller than the scope of my doing something on a machine. I've moved to just having a limited-use key with access to dev machines and git on the hard drive of each machine I use, which covers the cases I care about.

I restart my computer pretty infrequently and everytime I do, I can't connect to various things. Inevitably I realize I didn't run ssh-add and after debugging, things work again once I have run this command. But I never really looked into the basics.

Thanks to this article I realize why I need to do that every time. :)

SSH agent has other interesting possibilities too - it's an opaque channel from far server to your local machine, there must be various cool hacks you could do.

Years ago I wrote something to copy a file from a remote machine to my desktop over ssh-agent, worked well back then. https://matt.ucc.asn.au/ssh-xfer/

Thank you for the well done intro and the security risk implied with agent forwarding. I'd be curious how to actually sign with the ssh-agent as you mentioned it. I'd also like to know if when OS at login automatically decrypts the default private key, for later use e.g. pushing to github, what would be the risk? Would there be another preferred behaviour?

A pithy way of explaining all this: ssh-agent is exactly like a U2F token but implemented in software and using a slightly different protocol. But both do the same thing and serve the same purpose.

What I am missing is why ever use ssh agent, including any re-enforced variations, if ProxyJump has none of the ssh agent issues and has little to no other disadvantages?

ssh-agent is very useful, it's agent forwarding that is better handled with proxyjump. And my guess would be that forwarding came first and proxyjump only later.

ProxyJump doesn't help with the use cases cited in the tl;dr^Wintroduction.

1. Agent was designed to solve a larger problem; ProxyJump to solve a specialised subset of that problem.

2. The issues and disadvantages claimed with Agent Forwarding don't actually exist anymore: https://news.ycombinator.com/item?id=22753590

Cool, didn't know about -J! I have been using dodgy port forwarding all this time.

Now I just need to set up my Somu, and I will be living in 2020.

Nit: The TL;DR is what is called an introduction.

Yeah that is an odd choice of words.

I don’t think it’s a nit. It’s polite and proper feedback on the misuse of a term. The TL;DR here didn’t really convey anything useful from the rest of the post. So it ought to be rightly called “Introduction”, as you said.

Yep.

TL;DR = extreme executive summary

Intro = introduction

Same number of letters, but different meanings

It's not a big deal, but the armchair lawyer in me cringes and throws an OCD tantrum when spelling, grammar, semantics, and linguistic conventions are abused. "Weight at the busstop" or "u know what i mean thatswut im saying haha he even said it tiwce" :sigh: :meta-sigh:

I skip ssh-agent and use gpg-agent instead, which does both ssh and gpg agenting. Combined with monkeysphere, I store ssh private keys in gpg rather than in the encrypted ssh format.

I haven't heard about monkeysphere in years. Is it still being maintained?

Reply on news.ycombinator.com