The pitfalls of using SSH-agent, or how to use an agent safely | Hacker News Reader

The pitfalls of using SSH-agent, or how to use an agent safely

rabexc.org

180 pointskoolba68 comments

ObPlug for Guardian Agent, which is basically "safe" ssh-agent forwarding (and works with Mosh and SSH): https://github.com/StanfordSNR/guardian-agent

The basic story is that ssh-agent really just exposes a primitive of "please sign this challenge," which is useful locally, but the protocol wasn't designed to be forwarded. If requests are coming from a semi-trusted intermediary host, the protocol doesn't tell the agent (a) "what remote server is being authenticated to [i.e., who generated the challenge?]", or (b) "what command is going to be executed?" It doesn't even really know (c) "what (semi-trusted) host forwarded the challenge and is asking to authenticate to the remote server?" (although this one you can approximate today by running lots of agents and giving each local ssh command a different agent to forward requests to).

Guardian Agent is a sort of hack that allows the agent to know (a), (b), and (c) before deciding whether to grant or deny the request, so the user can set up policies like, "I'd like to allow `semi-trusted host x` to use <SSH identity i> to run "git pull from <repository name y>" when talking to `git server z`, but that's it." The basic ssh-agent protocol just doesn't have enough info to be able to do something like that.

But this doesn't solve the trust problem, though; only papers over it with a false sense of security.

The problem with traditional Agent Forwarding is that an intermediary may be compromised. A compromised intermediary can still fool Guardian Agent with false values for (a), (b), and (c). The only way Guardian Agent is more secure than unmodified ssh is that it's relatively unknown. Obscurity, however, isn't a defensible security perimeter.

No, a compromised intermediary can't fool Guardian Agent with false values for (a), (b), or (c). The client checks/enforces these. The details of how it works are in the linked paper.

From man ssh-add:

     -c      Indicates that added identities should be subject to confirmation be‐
             fore being used for authentication.  Confirmation is performed by
             ssh-askpass(1).  Successful confirmation is signaled by a zero exit
             status from ssh-askpass(1), rather than text entered into the re‐
             quester.

I'm a bit surprised that this is not mention in the article, as this seems very useful to make exploits more difficult.

Yep, or put this in your ~/.ssh/config

  AddKeysToAgent confirm

While helpful it mostly just says “ok to sign” and it’s super easy to accidentally say yes by hitting enter at the wrong time.

I need to look into the original post have wanted this for a while.

> 3. Only forward your agent connection to machines you trust.

You can get the convenience of agent forwarding without the negatives by using openssh's ProxyJump (or, in old versions ProxyCommand). Either allows you to transparently forward your ssh connection via another host (or chain of hosts).

ProxyJump can also be pretty magic. If you have machines all on a subnet (in my example below, 10.1.0.0/24) that you need to use the same SSH jumpbox to reach, you can easily set up a wildcard to transparently proxy jump to any of them:

  Host jumpbox-10.1.0.x
    HostName jumpbox.example.com
    Port ...
    User ...
  
  Host 10.1.0.*
    ProxyJump jumpbox-10.1.0.x

Then you can just run any random command like ssh 10.1.0.4 and it will just transparently jump through the jumpbox without you needing to specify it!

(Of course, if you have a local subnet that collides and you are also trying to SSH to local hosts then this won't be the right approach for you).

You can add exclusions for the hosts in your local colliding subnet with something like:

    Host 10.1.0.*
      ProxyJump jumpbox-10.1.0.x !host hosta,hostb.example.org,hostc,10.1.0.23

Interesting, would there be any reason to prefer that style over exempting them from the Host pattern?

  Host 10.1.0.* !hosta !hostb.example.org !hostc !10.1.0.23
    ProxyJump jumpbox-10.1.0.x

Doing it at the Host level would also make it easier if there were other config you wanted to apply for the group of machines (like IdentifyFile or Port)

Sorry, I made two mistakes when I wrote my response. The !host bit should be in the line above I've only used it in a Match block. I think your syntax is correct for a Host block exclusion.

e.g.,

    Match !exec "test1 -h %h -p %p >/dev/null 2>&1 || test2 >/dev/null 2>&1" host 10.1.0.*,*@example.org !host 10.1.0.4,10.1.0.11,hostb@example.org
        ProxyJump jumpbox-10.1.0.x

The tests can be, e.g., see if the destination host can be reached directly, and if yes, bypass the jumphost. Using %h and %p ssh will pass the destination hostname and port to the test command.

What use-case is there for agent forwarding? My work does have machines only accessible through bastion hosts, but I never find myself using agent forwarding. It's too easy to ProxyJump (e.g. like you mention, it works automatically thanks to .ssh/config) or if a direct connection between remote hosts is required, generate a new key (e.g. to use rsync).

My Yubikey requires physical touch to authorize an SSH connection so it doesn't matter if I forward my agent to malicious servers as their attempts to hijack my socket will result in timeouts from my not authorizing their use with my physical key

If you have a Mac with Touch ID, you can also use secretive, which requires fingerprint authentication:

https://github.com/maxgoedjen/secretive

Extra nice with the new Apple Magic Keyboard with Touch ID.

Yes, not using software keys is much the safest approach. They can't be stolen from your laptop oe remote forwarding.

Biometrics have their own set of problems.

Nice, I'd been looking for an alternative to sekey unfortunately.

Anything similar for Linux and Windows?

Modern SoCs do have secure enclaves, but how can one use it? I don't want a Yubikey dongle which i will forget at home when going to the office and vice versa. I'd be happy to get just the secure key storage without the fingerprint feature.

Yubikey has the Nano series[1][2] which you can leave in your laptop

[1]: https://www.yubico.com/gb/product/yubikey-5c-nano/

[2]: https://www.yubico.com/gb/product/yubikey-5-nano/

Would be nice but I'm an M1 Mac Mini user and don't have Touch ID. I wish they sold a Touch ID peripheral and not a full keyboard. I won't give up my hardware switches for Touch ID

unless they do it at the same time as you logging in to something else?

They'll only get one hop and I'll get a push notification immediately upon login to my server anyway

I'm in the process of setting this up for a network -- what are you using for push notifications?

First, there's no need to run a script to add keys on demand. Just enable "AddKeysToAgent yes" in your .ssh/config and have openssh > 7.2.

Second, you don't need a script to start the agent in your shell config, if you have systemd and logind. You can run one agent per session using systemd, with SSH_AUTH_SOCK set to a predictable path in /run/user/UID. Then you can include the predictable path as SSH_AUTH_SOCK in your .zshrc.

I thought the exact same thing, but by the dates on the comments, it looks like this article was written 7 years ago. I don't recall when I started using these features, but it might have been more recent than that, so I would say the omission is excusable.

First, the author writes as if shell servers with multiple people don't exist. Please see, for example, https://sdf.org/

Second, if you're going to exclude every server on which anyone else has root, you're essentially excluding all hosting (except colocation), since essentially that means all computers you don't physically manage.

There are many scenarios where a physically local computer is only trustworthy for the moment, and where this could change at any time. Windows computers, for instance, aren't trustworthy in the long term. Work computers may have nefarious software on them. Computers which are shared with multiple people (think of a family) may have users who aren't as careful.

A shell on a trustworthy provider such as SDF may be safer as a place to store keys and from which to run ssh-agent, so this scenario shouldn't be discounted.

Of course, this means nothing if your access to your shell on a trustworthy server is compromised, but wholesale discounting this is not helping people.

The author makes it pretty plain they do everything from a laptop that is in their sole control. I don't think it is a reasonable critique that every article is pertinent to every kind of user.

It's not written in the voice of, "this is good for most, but there are good reasons for other scenarios" as much as it's written in the voice of, "don't do this, don't do this".

And they lay the reasons why out in a compelling and thoughtful manner. Your own comment is written with the voice you are decrying. Personally, I find opinionated articles like TFA to be very helpful. And trying to write something that works as a global solution or insight is.. difficult. It is a significantly different task trying to produce something helpful or interesting with a limited scope (my usecase, my opinion, etc) vs something that is broad enough to touch on everything else. Its just hard to feel like you know enough to speak from that position, you know?

"Work computers may have nefarious software on them. Computers which are shared with multiple people (think of a family) may have users who aren't as careful."

Even a computer you manage yourself could have nefarious software on it, as most people don't usually manage their operating system and the operating system provider could serve up malicious software (intentionally or unintentionally), either originally or in an update. Same with the hardware of the machine, which could also compromise your keys or grant unauthorized people access.

I would not feel safe merely because I physically managed my own machines and knew something about security.

Really, a computer needs to be completely offline and with no ability to get online. Safest is not to use a computer at all.

I've been using keychain to manage my ssh-agent sessions forever and it works really well.

https://www.funtoo.org/Keychain

I like to use GPG-agent’s SSH agent emulation [1]. I start the agent with a systemd user service and have the keys expire after something like 10 minutes. A benefit is that it uses a GUI pinentry dialog to ask for your password if the requesting program doesn’t have stdin from a terminal, e.g. if I try to git push from Fugitive inside vim. I find that GPG-agent is more intuitive and less finicky than SSH-agent.

[1]: https://wiki.archlinux.org/title/GnuPG#SSH_agent

> Too many keys, github, and friends

This right here is what has been bugging me recently and the what is really selling me on the author's approach. Given that exposing a series of public keys to an SSH server acknowledges their connection, pervasive use of git hosting like Github/Gitlab, separating private and professional personas... How else do you keep maintain that consistently?

It's only mentioned in passing in the end but really, it's an underappreciated issue I haven't seen better solutions to.

Note that there is ssh-import-id[0] (and has a GitHub shortcut via ssh-import-id-gh) which can be useful when setting up new servers - it imports all your SSH keys stored in GitHub or another server. This is possible because your keys are public [1].

0: http://manpages.ubuntu.com/manpages/bionic/man1/ssh-import-i...

1: https://github.com/torvalds.keys

Why not use your PGP key for SSH authentication? Seems like there should be a way to integrate with Git, so that rather than having to specify the SSH command or setting up a complicated global config you can just make sure that your right GPG key is added to the project?

https://wiki.archlinux.org/title/GnuPG#Using_a_PGP_key_for_S...

There are a couple solutions to "too many keys".

The simplest and best solution is don't use SSH. For Git remotes using HTTPS, use personal access tokens / API tokens. For each repository you check out, change the git remote URL to include the username, like git remote add origin https://username@github.com/user/repo.git. Then configure Git to use your system's credential keychain and store your token there with the hostname and username. Or use a .netrc file. The correct access token will be selected based on the hostname and username.

If you do use SSH, you can configure a Git repo to use some specific SSH arguments to specify the key file, like git config core.sshCommand "ssh -i ~/.ssh/id_ed25519_clientX". Or you can configure your SSH client to have a custom Host entry for a fake host that specifies the SSH key to use, and use that fake host for your Git ssh remote URL.

Using hostnames like a.github.com and b.github.com and in your ~/.ssh/config and adding Host entries with IdentityFiles pointing to the right private keys is the cleaner solution imo.

How is this better than using IdentiesOnly and explicitly listing the one and only key that should be used for a given destination?

I've seen several systems that wrap the ssh binary to allow different agents to be forwarded to different hosts. Personally I think this is unwieldy -- it really should be something which is built into ssh directly.

Running multiple agents is also a bit ugly, especially if you are trying to consolidate your keys with an agent integrated with your desktop environment, which I think is the most common use case.

FWIW my proposal for fixing it is https://github.com/openssh/openssh-portable/pull/233 but it isn't the most elegant solution either I guess. It doesn't seem to have picked up much interest so I don't think it's likely to ever be merged (at least in its current form) which is fine. Hopefully some tamed version of agent forwarding appears directly in openssh someday, either as a simple key filter or something more complicated like guardian-agent

No need to wrap the binary for that, just something like this in ~/.ssh/config:

  Host *.foobar.com
    IdentityAgent ~/.ssh-agent-for-foobar.ssh

Of course, you still need to run a separate agent for each security domain you wish to keep separate.

Better:

    IdentitiesOnly yes # Only use the identity specified by IdentityFile instead of any presented by an agent
    ForwardAgent no    # Don't forward to a remote server.
    IdentityAgent none # Don't use an agent.
    AddKeysToAgent no # Don't add any unlocked keys to an agent.
    Host *.foobar.com
        IdentityFile ~/.ssh/path/to/private/key

> Of course, you still need to run a separate agent for each security domain you wish to keep separate.

Yes, and that's the rub.

It wouldn't be as bad if there was a way to make the ssh client manage the lifetime of the agent process itself (similar to ProxyCommand or something) And I definitely looked into building something like that.

Ultimately I thought that something built into the client was better overall for a couple reasons: * Filtering rules can be easily managed as .ssh/config settings * As I mentioned in the PR, I was able to reuse a lot of the existing code for the agent protocol that already is compiled into the client.

I also think the "filter" approach makes more sense than having truly separate ssh-agent binaries running. For one thing it's flexible to allow for multiple hops. Imagine if you are ssh'ing from A->B->C. On the first hop I want to give "B" access to keys "K1,K2" but then it wants to only give access to "K2" to C. With a protocol-filtering approach both hosts can prune back the amount of access being forwarded onwards.

"Never ever copy your private keys on a computer somebody else owns."

FTFY.

In more detail: if your private keys ever leave your computer via the network, it's a good idea to consider your private keys compromised and to burn them and create new ones.

If you're in an organization which uses SSH CAs and Principals (see https://dmuth.medium.com/ssh-at-scale-cas-and-principals-b27... for details), you'll only need to create a single keypair, get it signed, and you're good to go again.

> In more detail: if your private keys ever leave your computer via the network, it's a good idea to consider your private keys compromised and to burn them and create new ones.

Why such a rigid rule? What if I backup the private keys online, after encrypting them, using software that’s known to be secure? I guess my question is also about how to manage things when the private key stored locally is lost.

Are there yer any two-factor companion mobile apps that would increase SSH agent security locally (asks permission to use the agent key) or remotely (extra factor of authentication when login)?

I've seen PAM modules for otp/totp, but that really only adds security if authenticating against a remote machine, if you validate the second factor locally doesn't that still open you to mitm or other exploits? Can you trust your own environment?

For Mac users, there is this: https://github.com/TimidRobot/mac-ssh-confirm

Doesn't add 2FA so much as it prompts you for each agent use.

On the remote end, you can enable 2FA for logging into the server with libpam-google-authenticator.

There's also this to use a key in the Secure Enclave and Touch ID to confirm authentication: https://github.com/sekey/sekey

Duo also has a PAM module if you'd rather use a push notification https://duo.com/docs/duounix

https://www.stavros.io/posts/u2f-fido2-with-ssh/ goes over using FIDO2 devices with ssh, including storing the secret keys on the hardware. In that sense the passphrase becomes the second factor I guess.

You don't have to use agent forwarding to connect to servers behind a firewall! You can use SOCKS forwarding or static port forwarding in an ssh session to the bastion host, then a ProxyCommand option to tell the second ssh to tunnel over the first ssh.

However, it can still be useful to use different keys for different purposes; as the article explains, by default, ssh presents all your public keys to the remote server so it can identify you, which can be a privacy concern.

There is a -J option in recent ssh! `ssh -J machine1,machine2 machine3` It saves a lot of hassle with manually building the tunnels. The corresponding config option is ProxyJump.

However, I thought people were using agent when they really needed the "one remote computer logs into another" feature - e.g. for direct file transfer. I have never used SSH agent in my life, though - when I need something like that, I just generate fresh SSH keys on the remote computer.

Generating fresh ssh keys on the remote computer is an alternative to ssh-agent forwarding, not to ssh-agent per se. The alternative to ssh-agent per se is I think to leave your keys with no passphrase.

A timeout of 4 or 8 hours works well for me. Enough that it's useful, and part of my routine when I start my workday and return from lunch.

Is ssh-agent even worth the trouble?

Yes.

No. I have not found a single use case where an ssh-agent is worth the security problems it adds.

How do you push/pull a private GitHub repo on a remote machine you're SSHed into?

Deploy keys

So you leave a copy of a key that can push to your repo on the machine? How in the world is that supposed to be more secure than forwarding an agent that only authenticates you when you're actually connected? At least you don't leave a copy of the key itself on another machine.

Sorry, did not notice the push part. But it solves the pull part nicely.

Even for pulling I fail to see why having the deploy key sitting on the machine is supposed to be more secure than having it in your agent.

You have a key on a single machine that can only read a single repository with no additional hassle about securing the agent to not give out all your secrets at once, while it does not add any additional risk, as the remote machine already has full copy of the repo, so there is almost nothing to gain with the key if attacker had compromised the machine already.

If someone lifts the key they get perpetual access to your entire repo without any further need to have access to your machine. If they lift the repo then they get a snapshot of the branch(es) you have have in it at that moment in time - they'll need continued access to the machine if they want continued access to the data. There can be a nontrivial difference here right?

Being SSH'd into a machine doesn't matter.

You can create a new key _on that machine_ and authorize _that key_ for the specific purpose you need.

Or you can also just clone the repo on your local machine and add the remote machine as a git repo remote. Then you proxy all commits between github and the remote machine.

If you don't want to proxy commits between three repos (github, your local, your remote) then you could install sshfs on your local machine and sshfs-mount your remote machine.

Regardless; if you're running git on the remote machine then the root user of that remote machine would have access to the key. Why let them have access to _your_ key when they're only administering a machine for that repository? Let them have access to a shared key.

Use repo specific github oauth tokens with the least amount of permissions as possible

I would also add 'Always set a timeout with -t' both to protect the keys from misuse, and so you don't forget your passphrase when you rarely type it in.

    if [ "$?" == 2 ]; then

why not $? -eq 2?

why nested ifs?

Reply on news.ycombinator.com