Hacker News Reader

Comment by rvnx | Hacker News Reader

One safety tip: disable SSH Agent Forwarding before you connect, otherwise the remote server can theoretically reuse your private key to establish new connections to GitHub.com or prod servers (though this host is unlikely malicious).

https://www.clockwork.com/insights/ssh-agent-hijacking/ (SSH Agent Hijacking)

View on HN

The full command you want is:

    ssh -a -i /dev/null terminal.shop

to disable agent forwarding, as well as to not share your ssh public key with them, but that's just a little less slick than saying just:

    ssh terminal.shop

to connect.

I'm curious why you added `-i /dev/null`. IIUC, this doesn't remove ssh-agent keys.

If you want to make sure no keys are offered, you'd want:

  ssh -a -o IdentitiesOnly=yes terminal. Shop

I'm not sure if the `-i` actually prevents anything, I believe things other than /dev/null will still be tried in sequence.

Check for yourself with

    ssh -v -i /dev/null terminal.shop

    ssh -v terminal.shop

What you're looking for is that there is no line that says something like

    debug1: Offering public key: /Users/fragmede/.ssh/id_rsa RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Upon further testing, the full command you want is:

    ssh -a -i /dev/null -o IdentityAgent=/dev/null terminal.shop

to forcibly disable a local identity agent from offering up its identities as well, and not just agent forwarding.

Upon further testing,

    ssh -o IdentitiesOnly=yes terminal.shop

still offers up my public key on my system (macOS, OpenSSH_9.6p1, LibreSSL 3.3.6), contrary to what StackOverflow and the Internet seems to think. Tested by hitting whoami.filippo.io, linked in child comment.

Aha, yes, `-o IdentityAgent=/dev/null` is better for my intent. I was confused that `-i` wasn't removing .ssh/id_rsa from the candidates, but that was ssh-agent.

  ssh -a -i /dev/null -o IdentityAgent=/dev/null terminal.shop

That looks pretty solid. Thanks!

For a cool example (deanonymization), see https://words.filippo.io/dispatches/whoami-updated/ (discussed at time: https://news.ycombinator.com/item?id=34301768). Someone has crawled public keys from GitHub (tbh I was surprised that GitHub publishes them) and set up a database.

It's quite useful! I can give someone access to my server by grabbing their public key and creating an account for them, no need figure out how to send them the password to my server.

That's indeed how public keys are intended to work.

It's one of those obvious in hindsight things that gives me that "Internet was not a mistake" feels.

Gitlab does the same.

I've seen provisioning scripts and even cloud-init if I'm not wrong supporting downloading keys in that manner.

From one side it's cool from other side allows to bypass of system administrator for keys update more easily.

> You can make a search for all users, which will tell you there are 97,616,627 users at the time of this writing, but you can only fetch at most 1000 results from a search, and they don’t come in any clear order, so you can’t just make the next search start where the previous one left off (or I didn’t figure out how).

> What you can do though is request accounts created in a certain time range. If you get the time range right, so that it has less than 1000 entries, you can paginate through it, and then request the next time range.

This reminds me of when I tried to add a google drive storage backend to camlistore/perkeep (because I had nearly-unlimited free quota at the time). One of the things a perkeep blobserver needs to be able to do enumerate all the blobs it has, in order. You can send millions of blobs to google drive without issue, but you can't directly paginate a search for them in sorted order.

You could just issue a search for all blobs under your perkeep drive folder, keep paginating the result until you run out of pages, and then sort in memory, but there's really no way of knowing how many blobs you're going to end up with and you might blow out your blobserver's memory.

Perkeep blobs are identified by blobrefs, SHA sums of the contents of the blob, so they look like sha-[0-9a-f]{64}. Google drive lets you search for files with a name prefix, so you can search for like /perkeep/sha-* and see if the result has a pagination token (indicating that there are more than 1000 results), and if so then you search for each of /perkeep/sha-0*, /perkeep/sha-1*, ... , /perkeep/sha-f*, each time checking to see whether there are too many matches. When there's not too many matches, you've found the prefix length that will let you fetch a bounded number of blobrefs, emit them to the perkeep client, and then release the memory before fetching more.

  /pk/sha-\*          1000+ results (non-empty pagination token)
    /pk/sha-0\*       1000+ results (non-empty pagination token)
      /pk/sha-00\*    1000+ results (non-empty pagination token)
        /pk/sha-000\*  193  results,
                       sort these in memory and emit to client
        /pk/sha-001\*  179  results,
                       sort these in memory and emit to client
        ...
        /pk/sha-fff\*  223  results,
                       sort these in memory and emit to client

I didn't end up landing the patch before I lost interest, partly because it was pretty much the first golang I had tried writing. It was fun working out the above details, though.

> I tried to add a google drive storage backend to camlistore/perkeep (because I had nearly-unlimited free quota at the time)

This explains the quotas now :)

Offering your public key only allows them to identify the key and prove you have it. There is no security concern in sending this to an untrusted server.

Agent forwarding is a whole other beast.

Hm I thought I'd edited this. I was mistaken,

    ssh -o IdentitiesOnly=yes terminal.shop

works as expected, however I had an IdentityAgent set, and my key was being submitted via that route.

    ssh -o IdentitiesOnly=yes -o IdentityAgent=/dev/null terminal.shop

behaves as expected; same as

    ssh -a -i /dev/null -o IdentityAgent=/dev/null terminal.shop

Verified via whoami.filippo.io.

instructions not clear, my entire drive is empty now

1. Why is this something that would be enabled by default.

2. Can't you disable agent forwarding in a config file, so as not to have to clutter the command line?

I think it’s disabled by default on all distros I’ve used. You could add an entry to /etc/ssh_config or ~/.ssh/ if you want.

(It’ll still offer public keys by default in the exchange, but that’s “just” a privacy issue, not a privilege escalation problem.)

I just ran it in a `tmpfs` without any credentials:

    $ bwrap --dev-bind / / --tmpfs ~ ssh terminal.shop

I think you may want to clear the environment (e.g., of `SSH_AUTH_SOCK`) as well as isolate in a PID namespace as well. I also reflexively `--as-pid-1 --die-with-parent`.

    bwrap --dev-bind / / --clearenv --tmpfs ~ --unshare-pid --as-pid-1 --die-with-parent ssh terminal.shop

(The `bwrap` manpage says “you are unlikely to use it directly from the commandline,” yet I use it like this all the time. If you do, too, then we should be friends!)

Honestly the only thing that you need is -a (and only if you made the bad choice to do agent forwarding by default). Sending your pubkey (and a signature, because the server pretends to accept your pubkey for some reason?) isn't a security risk and you're (in theory) going to be providing much more identifying information in the form of your CC...

(And as the siblings mentioned this won't work to prevent your key from being sent if you're using an agent)

I agree with you, but there are those that take an extreme stance on privacy and I'm willing to oblige.

SSH Agent Forwarding does not happen by default. You need to include the -A option in your ssh command, unless maybe you've enabled it globally in your ~/.ssh/config file.

They can't get your private keys, but they could "perform operations on the keys that enable them to authenticate using the identities loaded into the agent" (quoting the man page). This would also only be possible while you are connected.

This is only a threat if you enable agent forwarding for all hosts.

If you enable agent forwarding for all hosts then yes, data will be forwarded.

Your link says:

> Don’t enable agent forwarding when connecting to untrustworthy hosts. Fortunately, the ~/.ssh/config syntax makes this fairly simple

Like you noted, ForwardAgent no is the default in /etc/ssh/ssh_config.

*disable ssh agent FORWARDING.

Which honestly should always be disabled. There are no trusted hosts.

That's baby+bathwater.

Just use ssh-add -c to have the ssh-agent confirm every use of a key.

TIL. Thanks! Gonna do wonders when working at places where I can't use a hardware key with physical confirmation of use.

My assessment still stands. Use proxyjump (-J) instead of proxy command whenever possible.

What can also help is specifying the right options right in ~/.ssh/config for certain hosts and domains: E.g. do "ForwardAgent no" globally, use a "Match *.my-trustworthy-company-domain.com" block and add "ForwardAgent yes" there.

Also very good for other options that are useful but problematic when used with untrustworthy target hosts, like ForwardX11, GSSAPIAuthentication, weaker *Algorithms (e.g. for those old Cisco boxes with no updates and similar crap).

Another neat trick is just using a ""Match *.my-trustworthy-company-domain.com" block" with an "IdentityFile ~/.ssh/secret-company-internal-key" directive. That key will then be used for those company-internal things, but not for any others, if you don't add it to the agent.

Whenever possible, yes, but AIUI it's not always possible; the one use case for which I believe full-on forwarding is required is using your personal credentials to transfer data between two remote servers (ex. rsync directly between servers). If there's a way to do that I would actually much appreciate somebody telling me, but I have looked and not found a way.

Or use a hardware backed ssh key you have to tap once for every use, like a Yubikey or Nitrokey.

Default for the last 24 years according to https://github.com/openssh/openssh-portable/blame/385ecb31e1...

I've found myself to be much more comfortable to just define all my private keys in ~/.ssh/config on a host-by-host basis.

AFAIK, this doesn't solve the SSH agent problem - the problem is the agent has access to all of those keys regardless of the host you connect to.

So forwarding your SSH agent means an administrator of the system you're connected to could use any of those host keys loaded in the agent to connect to their associated machine.

> There are no trusted hosts.

...your own (headless) server that's in the same room as you, when you're using your laptop as a thin-client for it?

Depending on what it's serving, and how up to date it is, and who else is on that network and can access the server, and who else can come into that same room when you're not there, and from where you get the software that you install on that server... it might be less trustworthy than you think.

But if that's your standard then the laptop you're connecting from is not trusted either, and then you're not even allowed to use your own keys.

You're allowed to draw sensible boundaries.

With all these recent exploits, I wouldn't even be 100% sure of that.

But if I can't trust even that host, I also can't trust the host I'm working on and which doesn't need agent forwarding to access my SSH agent.

Trusting one host is safer than trusting two hosts.

This is where certs are nice, sign one every morning with a 8/12 hour TTL

Interesting idea. Does need some automation though to make it practical irl.

Just to be clear, ssh agent forwarding is disabled by default and enabling it is always a hazard when connecting to machines that others also have access to.

Not at all specific to this.

Is it not standard practice to make different keys for different important services?

I have a private key for my prod server, a private key for GitHub, and a private junk key for authenticating to misc stuff. I can discard any without affecting anything else that's important.

If I authenticated with my junk key, would my other keys still be at risk?

> If I authenticated with my junk key, would my other keys still be at risk?

Yes, if you authenticate with your junk key (or no key), and SSH agent forwarding is enabled, you are still at risk. It lets the remote machine login to any server with any keys that are on your local SSH agent. Parent's link shows how this can be abused.

Fortunately, it's disabled by default, at least on newer versions.

It's a good practice, but it's somewhat against the grain of ssh defaults. It's not surprising that many people stick to the defaults.

It’s a practice, but not necessarily a standard one. In any case if even one person sees that, the advice will have served its purpose.

TIL, the good news I guess is I only ssh into my hosting platforms and GitHub who have a reason to protect my data since I pay them.

Still I'll be sure to break up my keys more going forward and disable SSH forwarding.

disabling agent forwarding is the important bit.

But if you do want to break up your keys more, make sure you specify IdentityFile and Identities Only in the per host definitions in your ssh config.

By default assuming you use an ssh agent (no forwarding) with multiple keys and a default ssh config, the behavior is to just try to auth with every key in order.

So if you're worried about the ssh server identifying you, you're still exposing yourself. I don't think this is much of a concern but worth noting.

Slightly more important: you're wasting time during the initial connection to fail authentication a few times. This can matter more with higher latency

Even more important: sshd has a configurable number of times a client is allowed to fail authentication in a session attempt. If you have too many other keys in your agent you will just fail to auth before it tries the key that is actually valid for that host.

The only reason/benefit for using different keys is to prevent someone from correlating your identity across different services... if you're worried about that go ham

If anything it's more standard practice to have agent forwarding disabled, since that's the default.

Default is disabled.

Exactly, this tip only applies if you reconfigured ssh to automatically forward agent to all hosts, which is absolutely insane.

I take it you mean disable ssh agent forwarding — the agent itself is fine. You should never forward your ssh agent to a box you don’t trust as much as your own.

Message edited, thank you, you are absolutely right.

And for privacy, don’t let it know your identity or username:

  ssh -o PubkeyAuthentication=no -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -a nobody@terminal.shop

Otherwise, the remote server can probably identify who you are on platforms like GitHub.

What I am reading from this there be dragons so don't use SSH to buy coffee!

I think so.

This feature is not enabled by default; "ForwardAgent = yes" has to be in the config file.

The article you cited makes it clear that you can turn this on for specific hosts in your private SSH config (and probably should do it that way).

So why wouldn't you?

Turning on forwarding globally and then having to remember to disable it for some untrusted hosts with -a looks silly and error-prone to me.

"ForwardAgent no" in ~/.ssh/config will do this automatically.

Not having "ForwardAgent yes" in ~/.ssh/config will do this automatically too.

Seems like a ridiculous amount of hoopla over something that isn't even a default.

Is "Host * \n AddKeysToAgent yes" acceptable from a security POV or should that also be per host?

Is it "yes" by default? If so, that seems insane given what the op said about it. But other comments say it's "no" by default. If it's "no" by default, why are people alarming us by bringing this up? And why for terminal.shop in particular?

The man page for ssh_config(5) says that it is set to "no" by default, at least on my computer.

Maybe there was some blanket advice in the past to enable it? Idk, this got me alarmed for nothing.

It's off by default. No idea what this fuzz is about. Gathering internet attention points maybe?

Using discoverable and non-discoverable keys via FIDO security keys will require PIN + physical confirmation, or just physical confirmation, by default if anyone tries to use your agent's keys.

If you want to use SSH forwarding reasonably safely, use a yubikey for ssh so you have to tap once for each hop. Now a MITM can't use your key for more hops without you physically consenting to each one.

That's terrifying. I don't understand why the design requires Forwarding to work without more explicit consent from the client at use time. (That is, when the middle tier wants to make a connection, it should forward an encrypted challenge from the server that can only be decrypted, answered, and re-encrypted by the original ssh keyholder on the client, similar to how, you know, ssh itself works over untrusted routers.

AFAIK, that’s exactly how agent forwarding works. The explicit part is that you need to explicitly turn it on

It is not the default, you would have to have a silly config for this to matter.

You can configure the agent to confirm each key usage to have your cake and eat it too. :)

It's also good to see if any malicious process tries to make use of the agent locally!

Thanks for the PSA. It gave me a good opportunity to double check that I hadn't enabled agent forwarding in any of my SSH scripts that don't need it.

You actually want to verify first or someone will mitm you, e.g. mitm.terminal.shop.rag.pub

With this one comment, you’ve convinced me that ssh apps are a bad idea

i usually just disable ssh agent forwarding globally by default, and only enable it selectively via my ~/.ssh/config

Dang. Didn't know this was a thing. Thank you!

here we go again. domain and path restricted cookies anyone?

Reply on news.ycombinator.com