Hacker News Reader

> If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

A better mandate would be to use ssh-agent. Using separate key pairs without an agent doesn't prevent key theft due to memory disclosure vulnerabilities like this one; it merely compartmentalizes the damage. But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities. Stealing a private key would require filesystem access or remote code execution, at which point all of your private keys are at risk, so you might as well just use a single private key per device.

For the best security, store your private key on a smartcard, so even an attacker with filesystem access or remote code execution can't steal it.

> But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities.

This is what I would assume, but has it been confirmed that this vulnerability doesn't affect identities provided by an SSH agent? If any of this "roaming" support was provided by the agent, it could be the case that the leakage can be triggered via SSH agent protocol requests. I doubt this is the case but I don't see anything confirming or denying it.

It would be nice to hear some official word on whether agent-based authentication is vulnerable, because if it isn't, the seriousness to me is greatly reduced, as I never load identities in the client directly.

The main issue identified by Qualys is a more generic memory disclosure bug:

https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-077...

This means they are basically able to dump the memory of just the running `ssh` process -- eerily similar to Heartbleed.

This means private keys stored by the `ssh-agent` process, outside of the `ssh` process connecting to an Evil Server(TM) are not affected.

This is because the protocol used between SSH Agent and SSH client does not transfer the entire private key, rather the SSH client asks the Agent to do a signing operation on it's behalf.

Good to know. Note regarding your final sentence, however: this design does not necessarily prevent the SSH agent from having a bug which can cause key disclosure via the agent protocol. It returns data buffers, so if it could be tricked into leaking key material into those data buffers, it may end up getting sent back to the server just the same (e.g. as a signature value). This would, however, be a bug in the agent and not the client.

The Qualsys analysis states that keys provided by an agent are not disclosed, and that encrypted keys are disclosed in their encrypted form.

https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-077...

Quoting from above: Finally, for these three reasons, passphrase-encrypted SSH keys are leaked in their encrypted form, but an attacker may attempt to crack the passphrase offline. On the other hand, SSH keys that are available only through an authentication agent are never leaked, in any form.

So if you use an agent, and follow the good advice to encrypt your private keys you should be safe(er).

See for older client versions:

http://martin.kleppmann.com/2013/05/24/improving-security-of...

or better for newer clients:

http://www.tedunangst.com/flak/post/new-openssh-key-format-a...

Ironically, ssh'ing into a random server is the exact thing you shouldn't do if you aren't patched.

Make sure you patch BEFORE doing something like ssh'ing into whoami.filippo.io "just because someone on the internet told me to"

This site is (from what I can tell) not malicicous, but copy pasting scripts and blindly following advice from the internet is a prime example of how an exploit would occur.

This may be a start:

  Host *
  IdentityFile ~/.ssh/%h

Comment: One key per host, named after the host you connect to

You also need to set

    IdentitiesOnly yes

if I remember the config setting correct. Note however that this only limits the offered keys during the authentication phase. If you use AgentForwarding, this still has the entire keyring available afterwards.

Don't use SSH agent forwarding.

https://heipei.github.io/2015/02/26/SSH-Agent-Forwarding-con...

which is not very helpful if you for example want to run git via ssh or similar commands on the target server.

There's places where there are two alternatives: agent forwarding or copy the key. Using ssh agent forwarding allows me to use a smart card for the key. It still has it's problems but it's much better than having the key on the remote machine - for example the use of a smartcard mitigates this vulnerability since the key never enters the process memory.

Not sure which exact scenario do you have in mind with ssh+git. ProxyCommand method works just fine for me with my private jumphost in the middle and github on the far end.

Note that for ProxyCommand to work, you don't need a full shell on a jumphost, just "AllowTcpForwarding yes" is enough. On the other hand, with AgentForwarding method you do need a full shell on a jumphost.

I have a vm that I use as development environment, my laptop is just a dumb terminal. I need to check out code there. It's not a jump host.

If you have clean network path from your terminal to the dev vm, why need for either ProxyCommand or ForwardAgent - just ssh to the vm directly, no?

Of course local, not forwarded ssh-agent on the terminal would be super-handy to avoid typing pass-phrase time and again; but that's different and independent from ForwardAgent.

I do ssh directly to the VM. It sits behind a VPN connection at aws. I need to make ssh connections from there to github. The key resides on a smartcard in my laptops usb slot. And that's where ssh-agent/ForwardAgent comes into play. I forward my local key to the remote VM.

Fair enough. Makes perfect sense, thanks!

Does anyone know how this interacts with ProxyCommand? If I fix this in ~/.ssh/config on my laptop, and I ssh to hostB through hostA using

  Host hostB
    ...
    ProxyCommand ssh hostA -W %h:%p

in the config on my laptop, do I also need to fix it on hostA?

CVE-2016-0777 doesn't interact with ProxyCommand in any special way. That said, after connecting to hostB with your example config above you will have 2 ssh sessions:

1) from your client to hostA

2) from your client to hostB

So to answer your question - no, vulnerable client on hostA is not a problem (or at least not in this particular use-case).

> Vulnerable client on hostA is not a problem.

Ok, that's what I was wondering. Thanks.

It is not my understanding that SSH will attempt to use only the file identified by IdentityFile, and yes, that is surprising now isn't it. If the server does not cooperate with you, your proposed configuration will give the server all of the keys.

Edit to add: While I was testing this understanding (which is correct) mioelnir's comment added the setting you need to get the behavior which I thought was automatic.

I have this:

    # IdentityFile magic, should be placed at very end of file
    Host *
    IdentityFile ~/.ssh/keys/id_ecdsa_%r@%h
    IdentityFile ~/.ssh/keys/id_rsa_%r@%h
    IdentityFile ~/.ssh/keys/id_ecdsa_ANY@%h
    IdentityFile ~/.ssh/keys/id_rsa_ANY@%h
    IdentityFile ~/.ssh/keys/id_ecdsa_%r@ANY
    IdentityFile ~/.ssh/keys/id_rsa_%r@ANY
    IdentityFile ~/.ssh/keys/id_ecdsa_ANY@ANY
    IdentityFile ~/.ssh/keys/id_rsa_ANY@ANY

What if my infrastructure is over 50 hosts? How am I going to distribute key files for 50 people with 50 keys every 14 days?

You could use something like ansible for the management of keys. Or you could centralize your authentication, which would probably be a lot more sane.

Even with central management, I don't know that I want to have 2500 keys floating around even if I have a management stack in place. That seems like an attack vector all in its own. Changing a key every other day on a 90 day rotation with 50 boxes. And fifty boxes isn't even that much. That's like 2 racks.

Even with config management this won't scale past about 2 or 3 people and 10-20 boxes.

Central auth is an option I guess but I think the better way to go would be a 2 factor with the key and hotp.

It's not that bad, it's part of the user data and should be provisioned the same way.

OpenSSH can also be used in a PKI fashion, where you use certificates instead of known_hosts and authorized_keys records. It's quite all right, but it comes with the same problems a full PKI does as you need to keep track of when the certificates expire. You also need a way to distribute CRLs so you still need configuration management.

The suggestion was not PKI though. I'd be happy with that. I already have a PKI in place. The suggestion was for a 1 key per box.

You automate it. At 50 hosts some automation like puppet or ansible, etc is worth the trouble, especially if it helps you make sure that all hosts have the correct keys on them - that's just basic security - make sure there are no keys that shouldn't be there.

What do you do now if someone leaves? Remove that person's key from all 50 hosts one at a time?

Or, at the very least, you use tmux with sync panes or csshx - and log into all 50 hosts at once, and then you can issue one rm / scp command if you are still doing things manually.

Well I have an automated credentials and authentication infrastructure in place. The point is that if I have 50 hosts (and my infrastructure has considerably more) and 3 employees with 50 keys each, there will be 2 key changes a day on a 90 day rotation.

Are my guys going to spend 5 minutes every morning making and pushing keys?

What if my hosts auto provision themselves and there are 5 new hosts every morning?

Am I going to make keys as part of infrastructure deployments and push them back to the workstations and update other peoples ssh configs?

I'm just saying if you've worked at scale, you'll realize that a key per box won't scale. I mean, anything can scale if you put enough effort into it. But the chances of disaster in lack of access or security breach, from over complication is way to high here.

Automating bad processes just makes it easier for them to fuck you.

If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

As patio11 notes, this would not at all be excessive as a response to the threat model, but even with an aggressive key rotation mandate, this still doesn't make your production infrastructure as secure as your corporate Confluence wiki with SSO. Your employee's private key is still a static credential, and any rotation policy (14 days? 30 days? 90 days?) will leave a significant window for an attacker to use a stolen credential. Additionally, using a single key per employee for all infrastructure access magnifies the attack surface of a stolen credential to everything you operate.

Full disclosure, I'm a co-founder at ScaleFT, a startup focused on solving these sorts of problems. We're releasing a patch to fully mitigate this for our users this morning.

I really hope that sysadmins/people are not using the same private key for work and pleasure. And for that matter, the same key at work for DEV/TEST/PROD. But you're probably right that a lot of them are.

As a sysadmin I am thankful I realized this was a mistake a long time ago, but you are right that I see it happen way too often for comfort.

Depends how you look at it. I use my key pair as my digital identification, whether at work or at home. I only have the one set. That logic held up until now I suppose, with the leaking of private keys.

For the at-work keys, I differentiate between work/workstation and work/laptop. Full disk encryption and the bcrypt based ssh private key format hopefully buy me enough time to revoke the laptop keys and have a KRL update pushed to all servers if it gets stolen.

It does depend. This may be leftover behavior from my contract days, but I've always had at least one key per entity with which I interact[1].

The reasoning is that it is far easier for me (or the entity, but that's a bit different) to delete one keyfile to sever access than it is for me to rekey everything else. I don't want access to things I'm not actively engaged with - compromises happen, even to engineers' laptops, and that conversation with former employers is too much like calling up your exes to tell them about a VD test result for my comfort.

As far as it being digital identification, lots of companies have IDs separate from your DL/passport. This is usually because the company ID provides access to something your other ID doesn't. Same principle.

[1] There are lots reasons to have lots of different keys, and only having one per entity is pretty rare for me.

If you want to stop this, require a specific, non-standard, RSA key size for each access group. They'll have to generate new keys. For ECC, maybe ssh certificates will work?