OpenSSH: client bug CVE-2016-0777 | Hacker News Reader

OpenSSH: client bug CVE-2016-0777

undeadly.org

617 pointsmoviuro215 comments

Use the workaround immediately and patch ASAP, both on your workstation and across your infrastructure where SSH is used, which can be in surprising places.

SSH is designed so that, even if you connect to an evil host, the host only learns your public key, not your private key. This leaks your private key to an evil host. You might think "I only SSH into boxes that I own, so I'm good, right?", but if you in the future lose a box to the enemy, they can then use that box to grab your private key, and then use that key to get into every other box you can access with that keypair.

This turns your sysadmin's photo blog hosted on Digital Ocean into a potential vector into your entire infrastructure, since there is a high likelihood they use the same private key for both.

If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

You should expect adversaries to add an exploit for this to their rootkits. That would be bad enough since attackers targeting your infrastructure may exist, but this is also plausibly exploitable in a spray-and-pray fashion. Root any server on the Internet, using e.g. a WordPress vulnerability. Install key-stealing malicious SSH server. Forward all keys to your C&C infrastructure. Your infra does a reverse lookup from public key to Github account (trivial -- try "ssh whoami.filippo.io" if you don't believe me) and immediately optimistically tries to use the stolen key to log into the web tier of any site listed in their email/bio/Twitter profile/etc, pinging the attacker when it succeeds.

> If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

A better mandate would be to use ssh-agent. Using separate key pairs without an agent doesn't prevent key theft due to memory disclosure vulnerabilities like this one; it merely compartmentalizes the damage. But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities. Stealing a private key would require filesystem access or remote code execution, at which point all of your private keys are at risk, so you might as well just use a single private key per device.

For the best security, store your private key on a smartcard, so even an attacker with filesystem access or remote code execution can't steal it.

> But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities.

This is what I would assume, but has it been confirmed that this vulnerability doesn't affect identities provided by an SSH agent? If any of this "roaming" support was provided by the agent, it could be the case that the leakage can be triggered via SSH agent protocol requests. I doubt this is the case but I don't see anything confirming or denying it.

It would be nice to hear some official word on whether agent-based authentication is vulnerable, because if it isn't, the seriousness to me is greatly reduced, as I never load identities in the client directly.

The Qualsys analysis states that keys provided by an agent are not disclosed, and that encrypted keys are disclosed in their encrypted form.

https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-077...

Quoting from above: Finally, for these three reasons, passphrase-encrypted SSH keys are leaked in their encrypted form, but an attacker may attempt to crack the passphrase offline. On the other hand, SSH keys that are available only through an authentication agent are never leaked, in any form.

So if you use an agent, and follow the good advice to encrypt your private keys you should be safe(er).

See for older client versions:

http://martin.kleppmann.com/2013/05/24/improving-security-of...

or better for newer clients:

http://www.tedunangst.com/flak/post/new-openssh-key-format-a...

The main issue identified by Qualys is a more generic memory disclosure bug:

https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-077...

This means they are basically able to dump the memory of just the running `ssh` process -- eerily similar to Heartbleed.

This means private keys stored by the `ssh-agent` process, outside of the `ssh` process connecting to an Evil Server(TM) are not affected.

This is because the protocol used between SSH Agent and SSH client does not transfer the entire private key, rather the SSH client asks the Agent to do a signing operation on it's behalf.

Good to know. Note regarding your final sentence, however: this design does not necessarily prevent the SSH agent from having a bug which can cause key disclosure via the agent protocol. It returns data buffers, so if it could be tricked into leaking key material into those data buffers, it may end up getting sent back to the server just the same (e.g. as a signature value). This would, however, be a bug in the agent and not the client.

Ironically, ssh'ing into a random server is the exact thing you shouldn't do if you aren't patched.

Make sure you patch BEFORE doing something like ssh'ing into whoami.filippo.io "just because someone on the internet told me to"

This site is (from what I can tell) not malicicous, but copy pasting scripts and blindly following advice from the internet is a prime example of how an exploit would occur.

This may be a start:

  Host *
  IdentityFile ~/.ssh/%h

Comment: One key per host, named after the host you connect to

I have this:

    # IdentityFile magic, should be placed at very end of file
    Host *
    IdentityFile ~/.ssh/keys/id_ecdsa_%r@%h
    IdentityFile ~/.ssh/keys/id_rsa_%r@%h
    IdentityFile ~/.ssh/keys/id_ecdsa_ANY@%h
    IdentityFile ~/.ssh/keys/id_rsa_ANY@%h
    IdentityFile ~/.ssh/keys/id_ecdsa_%r@ANY
    IdentityFile ~/.ssh/keys/id_rsa_%r@ANY
    IdentityFile ~/.ssh/keys/id_ecdsa_ANY@ANY
    IdentityFile ~/.ssh/keys/id_rsa_ANY@ANY

It is not my understanding that SSH will attempt to use only the file identified by IdentityFile, and yes, that is surprising now isn't it. If the server does not cooperate with you, your proposed configuration will give the server all of the keys.

Edit to add: While I was testing this understanding (which is correct) mioelnir's comment added the setting you need to get the behavior which I thought was automatic.

You also need to set

    IdentitiesOnly yes

if I remember the config setting correct. Note however that this only limits the offered keys during the authentication phase. If you use AgentForwarding, this still has the entire keyring available afterwards.

Does anyone know how this interacts with ProxyCommand? If I fix this in ~/.ssh/config on my laptop, and I ssh to hostB through hostA using

  Host hostB
    ...
    ProxyCommand ssh hostA -W %h:%p

in the config on my laptop, do I also need to fix it on hostA?

CVE-2016-0777 doesn't interact with ProxyCommand in any special way. That said, after connecting to hostB with your example config above you will have 2 ssh sessions:

1) from your client to hostA

2) from your client to hostB

So to answer your question - no, vulnerable client on hostA is not a problem (or at least not in this particular use-case).

> Vulnerable client on hostA is not a problem.

Ok, that's what I was wondering. Thanks.

Don't use SSH agent forwarding.

https://heipei.github.io/2015/02/26/SSH-Agent-Forwarding-con...

which is not very helpful if you for example want to run git via ssh or similar commands on the target server.

There's places where there are two alternatives: agent forwarding or copy the key. Using ssh agent forwarding allows me to use a smart card for the key. It still has it's problems but it's much better than having the key on the remote machine - for example the use of a smartcard mitigates this vulnerability since the key never enters the process memory.

Not sure which exact scenario do you have in mind with ssh+git. ProxyCommand method works just fine for me with my private jumphost in the middle and github on the far end.

Note that for ProxyCommand to work, you don't need a full shell on a jumphost, just "AllowTcpForwarding yes" is enough. On the other hand, with AgentForwarding method you do need a full shell on a jumphost.

I have a vm that I use as development environment, my laptop is just a dumb terminal. I need to check out code there. It's not a jump host.

If you have clean network path from your terminal to the dev vm, why need for either ProxyCommand or ForwardAgent - just ssh to the vm directly, no?

Of course local, not forwarded ssh-agent on the terminal would be super-handy to avoid typing pass-phrase time and again; but that's different and independent from ForwardAgent.

I do ssh directly to the VM. It sits behind a VPN connection at aws. I need to make ssh connections from there to github. The key resides on a smartcard in my laptops usb slot. And that's where ssh-agent/ForwardAgent comes into play. I forward my local key to the remote VM.

Fair enough. Makes perfect sense, thanks!

What if my infrastructure is over 50 hosts? How am I going to distribute key files for 50 people with 50 keys every 14 days?

You automate it. At 50 hosts some automation like puppet or ansible, etc is worth the trouble, especially if it helps you make sure that all hosts have the correct keys on them - that's just basic security - make sure there are no keys that shouldn't be there.

What do you do now if someone leaves? Remove that person's key from all 50 hosts one at a time?

Or, at the very least, you use tmux with sync panes or csshx - and log into all 50 hosts at once, and then you can issue one rm / scp command if you are still doing things manually.

Well I have an automated credentials and authentication infrastructure in place. The point is that if I have 50 hosts (and my infrastructure has considerably more) and 3 employees with 50 keys each, there will be 2 key changes a day on a 90 day rotation.

Are my guys going to spend 5 minutes every morning making and pushing keys?

What if my hosts auto provision themselves and there are 5 new hosts every morning?

Am I going to make keys as part of infrastructure deployments and push them back to the workstations and update other peoples ssh configs?

I'm just saying if you've worked at scale, you'll realize that a key per box won't scale. I mean, anything can scale if you put enough effort into it. But the chances of disaster in lack of access or security breach, from over complication is way to high here.

Automating bad processes just makes it easier for them to fuck you.

You could use something like ansible for the management of keys. Or you could centralize your authentication, which would probably be a lot more sane.

Even with central management, I don't know that I want to have 2500 keys floating around even if I have a management stack in place. That seems like an attack vector all in its own. Changing a key every other day on a 90 day rotation with 50 boxes. And fifty boxes isn't even that much. That's like 2 racks.

Even with config management this won't scale past about 2 or 3 people and 10-20 boxes.

Central auth is an option I guess but I think the better way to go would be a 2 factor with the key and hotp.

It's not that bad, it's part of the user data and should be provisioned the same way.

OpenSSH can also be used in a PKI fashion, where you use certificates instead of known_hosts and authorized_keys records. It's quite all right, but it comes with the same problems a full PKI does as you need to keep track of when the certificates expire. You also need a way to distribute CRLs so you still need configuration management.

The suggestion was not PKI though. I'd be happy with that. I already have a PKI in place. The suggestion was for a 1 key per box.

If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

As patio11 notes, this would not at all be excessive as a response to the threat model, but even with an aggressive key rotation mandate, this still doesn't make your production infrastructure as secure as your corporate Confluence wiki with SSO. Your employee's private key is still a static credential, and any rotation policy (14 days? 30 days? 90 days?) will leave a significant window for an attacker to use a stolen credential. Additionally, using a single key per employee for all infrastructure access magnifies the attack surface of a stolen credential to everything you operate.

Full disclosure, I'm a co-founder at ScaleFT, a startup focused on solving these sorts of problems. We're releasing a patch to fully mitigate this for our users this morning.

I really hope that sysadmins/people are not using the same private key for work and pleasure. And for that matter, the same key at work for DEV/TEST/PROD. But you're probably right that a lot of them are.

Depends how you look at it. I use my key pair as my digital identification, whether at work or at home. I only have the one set. That logic held up until now I suppose, with the leaking of private keys.

It does depend. This may be leftover behavior from my contract days, but I've always had at least one key per entity with which I interact[1].

The reasoning is that it is far easier for me (or the entity, but that's a bit different) to delete one keyfile to sever access than it is for me to rekey everything else. I don't want access to things I'm not actively engaged with - compromises happen, even to engineers' laptops, and that conversation with former employers is too much like calling up your exes to tell them about a VD test result for my comfort.

As far as it being digital identification, lots of companies have IDs separate from your DL/passport. This is usually because the company ID provides access to something your other ID doesn't. Same principle.

[1] There are lots reasons to have lots of different keys, and only having one per entity is pretty rare for me.

For the at-work keys, I differentiate between work/workstation and work/laptop. Full disk encryption and the bcrypt based ssh private key format hopefully buy me enough time to revoke the laptop keys and have a KRL update pushed to all servers if it gets stolen.

If you want to stop this, require a specific, non-standard, RSA key size for each access group. They'll have to generate new keys. For ECC, maybe ssh certificates will work?

As a sysadmin I am thankful I realized this was a mistake a long time ago, but you are right that I see it happen way too often for comfort.

Does it strike anyone else as bizarre / poor form for an experimental feature to be enabled by default in OpenSSH, which is normally very conservative with option defaults?

What's more, it was not documented in the ssh_config(5) man page. If it had been, I probably would have disabled it long ago when hardening my SSH config.

Time to head to the source to look for other undocumented options...

Update: my findings are here (scroll to bottom for the upshot): https://gist.github.com/AGWA/e92d4f5343be1f7a941d

UseRoaming is the only one to be concerned about. There are many other undocumented options, but they're all aliases for a documented option or are deprecated/unsupported.

https://github.com/openssh/openssh-portable/blob/e6c85f8889c... Here you go

Actually, I quickly generated a list of undocumented config options: http://sprunge.us/QDSE

Edit: Fixed version http://sprunge.us/LVYB

most of those seem to be documented: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/...

I did a quick comparison between the OpCodes enum and the latest man page, and `UseRoaming` is the only real undocumented option.

Weird, my manpages didn't get updated despite man claiming they did. Fixing that now.

Edit: fixed, http://sprunge.us/LVYB

I believe "kerberosauthentication" and "rhostsauthentication" are documented as well. I'd check the "official" OpenBSD man page instead of whatever you have on your system. Regardless, thanks.

In the latest OpenSSH-portable source, those are considered "unsupported" and "deprecated," respectively (which means they're ignored, essentially). See https://gist.github.com/AGWA/e92d4f5343be1f7a941d#file-undoc... as well as the source https://github.com/openssh/openssh-portable/blob/e6c85f8889c...

Neither of those are mentioned on the latest manpage from git.

Please report back what you find!

It just sounds like a big code-base fuckup. They more or less admit it: "Server side was disabled/gutted for years already, but this aspect was surprisingly forgotten."

Sounds like this was put in at one time, forgotten about, and the code lingered for a long time until someone pointed it out. SSH as a protocol is pretty crazy. Everyone loves it, but its a lot of things in one, which ironically goes against the unix philosophy. Its a remote terminal, a file transfer server, a network tunnel server, a socks server, etc. There's a lot of stuff in there and I imagine difficult to work with sometimes.

No word if this is enabled in Putty, but I imagine it is if its using openssh libraries.

Given that this exploit happens during capability negotiation (or whatever SSH calls that part of the protocol), it also cannot be mitigated via pledge[2], which is where OpenBSD has been focusing a lot of attention. This is an unusual stumble for the OpenBSD team. Client-side privsep support, if such a thing existed, might mitigate attacks like this. As it stands protecting against exploits of this type wasn't even on the hardening roadmap.

http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/...

Related: "Protecting sshd using spiped" by cperciva

http://www.daemonology.net/blog/2012-08-30-protecting-sshd-u...

SCP/SFTP is actually a separate binary.

PuTTY is definitely not affected since it has its own SSH implementation.

PuTTY is completely different code.

Pretty sure this is OpenSSH only. PuTTY and SecureCRT, at least, are reported not to be affected.

We use WinSCP also, a file copying utility; it would be useful to know whether that's affected if anybody has that info.

Last time I looked WinSCP used PuTTY components or a fork. Would appreciate additional verification of this, though.

Server-side you can see the "full config" sshd uses:

  $ /usr/sbin/sshd -T -f /etc/ssh/sshd_config

Modify, if necessary, to point to the location of your `sshd` and configuration file.

I came here to ask exactly this, and also why this feature isn't documented. Sounds like a nice little exploit.

Someone tweeted this, the roaming code doesn't seem to be very well managed: https://twitter.com/marver/status/687644904575627264

Edit 1: Here's some relevant commits too https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2

Edit 2: This mailing list post seems to discuss the vulnerable feature http://www.gossamer-threads.com/lists/openssh/dev/49018?do=p...

Edit 3: Got a better description of actual impact of the bugs:

>Experimental roaming code in the ssh client could be tricked by a hostile sshd

>server, potentially leaking key material. CVE-2016-077 and CVE-0216-078.

>Prevent this problem immediately by adding the line "UseRoaming no" to

>/etc/ssh/ssh_config.

> Someone tweeted this, the roaming code doesn't seem to be very well managed: https://twitter.com/marver/status/687644904575627264

The subsystem was apparently experimentally (hence the undocumented option) introduced 6 years ago and essentially never used, the server support was never implemented: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/roa...

So the real question is, can a MITM intercept connections to boxen you frequent to exploit this? Or is it limited to connecting to hostile honeypots?

Apparently not possible with MITM.

"The authentication of the server host key prevents exploitation by a man-in-the-middle, so this information leak is restricted to connections to malicious or compromised servers."

https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-Ja...

... and those that you haven't established TOFU with yet

Tested quickly, doesn't look like it. Host key checking happens before the roaming stuff.

That doesn't mean it's limited to hostile honeypots, if it turns out that this is easily exploitable it'll be a complete shitstorm with sysadmins getting owned left and right with rootkits implementing automatic exploitation.

What is "boxen"?

Plural of UNIX (or equivalent) system [1]

[1] http://www.catb.org/jargon/html/B/boxen.html

> >server, potentially leaking key material. CVE-2016-077 and CVE-0216-078.

Presumably you meant CVE-2016-0777 and CVE-2016-0778

I'm just quoting OFFICIAL SOURCES here.

Workaround (yes, it's client-side):

  # echo -e "Host *\n\tUseRoaming no\n" >> /etc/ssh/ssh_config

Disclaimer: won't work on all operating systems, shells, etc. YMMV. Consult a doctor before following any advice you get from the Internet. Void where prohibited. Restrictions may apply.

Edited per comments below

Don't ever fix your `ssh_config` by appending stuff to the end of the file. The configuration syntax allows for block constructs without an explicit end marker (like `Match` and `Host`). Appending will cause all kinds of sadness.

It's from the linked page.

Indeed. The page gives bad advise. If your config has `Host` blocks, like often in people's personal configs, or if you have a `Match` block, the new directive only applies to the last of those blocks in the config file.

Sufficient to place it at the top of ~/.ssh/config? Or does it also need a block header of its own, like:

    Host *
    UseRoaming no

I confirmed that it does not need to be placed into a Host block, as long as it is above all of your blocks.

For instance:

   UseRoaming no

   Host *
      Blah yes

Test with: ssh -v remote.ssh.host.com uptime 2>&1 | grep -i roaming

If it returns nothing, the config fix is active. If it isn't active, you'll see 'debug1: Roaming not allowed by server'

Which is why they're appending a newline and Host *?

Note that the example was changed as a result of my initial comment.

Presumably you would have to connect to a malicious host to be effected? Or perhaps a MITM on your connection to a legit host can exploit you somehow.

From the updated OP:

> The authentication of the server host key prevents exploitation by a man-in-the-middle, so this information leak is restricted to connections to malicious or compromised servers.

Malicious, compromised, or new servers. Because really, how many people check the host key for a newly spun up EC2 instance?

> Presumably you would have to connect to a malicious host to be effected?

Malicious or compromised.

> roaming code in the ssh client could be tricked by a hostile sshd server, potentially leaking key material.

Would this work too?

  $ echo "UseRoaming no" >> ~/.ssh/config

Yes, but it'll only affect that user.

And only the last `Host` entry in that config, if present.

Yes, but it would not apply to other users.

I'm emailing the one-liner to folks and found it helpful to just include the root subshell:

    sudo bash -c 'echo -e "Host *\n\tUseRoaming no\n" >> /etc/ssh/ssh_config'

If you have a mac (Yosemite), it looks like you want to add " UseRoaming no" under the "Host *" line in /etc/ssh_config (as root). You can test it before and after with this:

ssh -v -T git@github.com 2>&1 | grep Roaming

debug1: Roaming not allowed by server <- bad

ssh -v -T git@github.com 2>&1 | grep Roaming

(no output is good)

On El Capitan the file is /etc/ssh/ssh_config

On Mountain Lion it's /private/etc/ssh_config.

/etc points to /private/etc on OS X, so it's all good.

A few of my friends found multiple copies of ssh_config lying around, so I wrote this one-liner that looks at 3 separate known ssh_config files & patches any vulnerable ones that it finds.

$ for SSH_CONF in /etc/ssh/ssh_config /etc/ssh_config /private/etc/ssh_config; do [ -f $SSH_CONF ] && ! grep -q 'UseRoaming no' $SSH_CONF && echo "Patching $SSH_CONF" && echo -e '\nHost *\n UseRoaming no' | sudo tee -a $SSH_CONF > /dev/null; done

Here is a user-friendly script that will disable UseRoaming on your os x computer. If you want to send this out to your coworkers, remind them that they have to right-click and choose open to execute a .command script (unless they have "allow apps downloaded from anywhere" enabled)

https://gist.github.com/logicalmethods/49f42190406667cbbbee

Interestingly I haven't been able to get the Roaming not allowed message to show on my client with or without the config option set.

Did you copy the whole line as "g ssh -v -T git@github.com 2>&1 | grep "Roaming debug1: Roaming not allowed by server"" ?

I misread GP's command and made that mistake. The correct command is :

ssh -v -T git@github.com 2>&1 | grep "Roaming"

My bad, the comment system stripped my newlines. Fixed the post.

It looks like arch just got the fix in really quick. I didn't recall updating my box this morning but I must have, because I already had the patched version in place.

Testing the ssh client config workaround:

   ssh -v user@localhost 2>&1 >/dev/null | grep -i 'roaming'

returns "debug1: Roaming not allowed by server" when vulnerable, and nothing when not. YMMV, only tested on a few machines, etc.

does this require a ssh server running on localhost?

Yes, that command is connecting to an ssh server on localhost, but you could connect to any ssh server that you trust...

  ssh -v -T git@github.com 2>&1 | grep -i 'roaming'

Since this is a client side issue, can this be used to exploit those automated scanners who try to break into your SSH machine?

Authenticated scanners that use key auth like Qualys' security appliances could have private keys that are valid across the organization, and if using an affected client version, could leak this information to a malicious system on your network.

No. The scanners are looking for password-accessible accounts, not keyed accounts. The scanners won't have useful keys, nor listening ssh daemons.

Release is here, with many more details, and other security issues fixed: https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-Ja...

Details: http://www.openwall.com/lists/oss-security/2016/01/14/7

Does anybody else find this eerily reminiscent of heartbleed?

Obscure broken feature that nobody uses (or needs) is enabled by default and allow private keys to leak.

It is OpenSSH, the clown car of bugs in obscure features nobody uses.

Does anybody know if LibraSSL has released an SSH client yet? I can't code well enough to make security code safe, so I won't be able to help them out on it, but it sounds more and more like OpenSSH can't either.

You seem confused. The LibreSSL people released a SSH client years ago, called OpenSSH. Which is not the same as OpenSSL, which suffered Heartbleed.

I can't blame him, considering that people have been encouraging this sort of attitude towards the OpenSSL maintainers. It's time they tasted their own medicine.

My hardened ~/.ssh/config on OS X 10.11:

  Host *
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
    HostKeyAlgorithms ssh-ed25519,ssh-rsa
    ChallengeResponseAuthentication no
    UseRoaming no

If you only connect to newer servers you can further restrict ciphers to only use AEADs (only list the chacha20-poly1305 and aes-gcm ciphers). I assume using AEADs-only makes the MACs keyword obsolete, is this correct?

Config is based on tips from https://stribika.github.io/2015/01/04/secure-secure-shell.ht...

> I assume using AEADs-only makes the MACs keyword obsolete, is this correct?

Correct. You can test this by running SSH with -v/--verbose and observing these log lines:

    debug1: kex: server->client aes256-gcm@openssh.com <implicit> none
    debug1: kex: client->server aes256-gcm@openssh.com <implicit> none

When using a cipher that does not support AEAD, the 2nd field will include the MAC instead of <implicit>

    debug1: kex: server->client aes256-ctr hmac-sha2-256-etm@openssh.com none
    debug1: kex: client->server aes256-ctr hmac-sha2-256-etm@openssh.com none

I wasn't sure what the 3rd "none" field is, but after some digging it appears to be the compression algorithm: https://github.com/openssh/openssh-portable/blob/master/kex....

And I confirmed this with ssh -C:

    debug1: kex: server->client aes256-ctr hmac-sha2-256-etm@openssh.com zlib@openssh.com
    debug1: kex: client->server aes256-ctr hmac-sha2-256-etm@openssh.com zlib@openssh.com

Mozilla also maintains a SSH guide for both clients and servers: https://wiki.mozilla.org/Security/Guidelines/OpenSSH

Funny story, I erroneously threw `UseRoaming no` on my server config and freaked the fuck out when I couldn't get into via SSH.

Six rescue sessions later, I figured out it shouldn't have been there ...

In the future, I'm definitely going to read documentation & test config files (sshd -t -f /path/to/sshd_config, in this case)

EDIT: Moral of the story, I am not a smart person sometimes.

Having been through some difficult times when i had been let down by everyone around me, i came across Brandon Reid who is a hacker its actually impossible to put in words how much of a Genius he is and also cant stop thanking him for helping me through my divorce, and also my Nephews grades. He is a Blackhat hacker and very capable of almost and everything, he is actually one of the best out there and also shows you proof before charging you, in my own case the money wasnt the problem and i can gladly say every money spent was so worth it. I have reffered a number of people who will prefer not to be named to Brandon and all have been immensly satisfied with the Top Notch hacking service Brandon offers.To whoever who is lucky enough to read this its a new year and i am only doing this for those genuine people out there who would want the services of a hacker please as i said before be careful how you speak to him his a BigFish. He can be contacted on his services email Brandonreid001@gmail.com or text +1 813 379 2141 . Do mention the name Jade

Roaming feature has been in OpennSSH code for a while, but is undocumented: http://superuser.com/a/826734/47771

"Affects all OpenSSH 5.4 - 7.1". 5.4 was released March 2010 it seems

AFAIK the roaming feature has been disabled on the server side for a while, but mistakenly left enabled on the client.

Seems correct: https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2

How does a client bug end up leaking private key material to the server if the private key is in an entirely different process (the agent)? Or is this only an issue if you are loading your identity directly in the client?

I would love to hear from someone who knows the answer to this. Does this affect ssh-agent? I am rotating my keys regardless, but I am curious.

According to: https://news.ycombinator.com/item?id=10902833 it does not:

> Finally, for these three reasons, passphrase-encrypted SSH keys are leaked in their encrypted form, but an attacker may attempt to crack the passphrase offline. On the other hand, SSH keys that are available only through an authentication agent are never leaked, in any form.

So, if there was forgotten code in OpenSSH (which, how the fuck does that happen?), where else might there be forgotten code?

Also: this is a really great example of why you should never have undocumented features in your code.

While the bug doesn't expose a user to mitm attacks in general, am I correct in thinking that folks using "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" have basically opened the door for that?

https://github.com/search?q=-o+UserKnownHostsFile%3D%2Fdev%2...

Ughhh, that's insane.

> Ughhh, that's insane.

Not really. This is for deploy systems which deploy to a trusted environment (for instance through VPN, network security etc.).

Even if you are using some kind of prebaked images to deploy, you should be generating individual keys using the SSH PKI features per machine as part of your individual host configurations.

This allows you to verify hosts while having never seen their keys. Just totally shutting off verification is a horrible idea.

If someone can actively MITM in your network you have other problems.

That argument has been shown to be nonsense for years, it can apply to any mitigation technology or privilege reduction technique. The point is to reduce the amount of harm that can be done, because "just write software without bugs" isn't a solution at all.

> That argument has been shown to be nonsense for years, it can apply to any mitigation technology or privilege reduction technique. The point is to reduce the amount of harm that can be done, because "just write software without bugs" isn't a solution at all.

I don't disagree with this, but if you can MITM my traffic then impersonating SSH is the least of my worries. The chance that I will randomly SSH into a machine is pretty small to begin with whereas the deployment tools themselves for instance will push out code changes in regular intervals throughout the cluster. My point is: if you can actively MITM my traffic or anything similar in severity, then there are much more interesting targets than SSH.

As far as I know, this will affect any OSX, am I right ?

It affects all operating systems: http://marc.info/?l=openbsd-tech&m=145278077820529&w=2

Is there any way to update openssh on my side, without waiting to apple ? Is this enough ?https://mochtu.de/2015/01/07/updating-openssh-on-mac-os-x-10...

Even easier, just place this text:

    UseRoaming no

on its own line in your ~/.ssh/config file. If you don't have such a file, create it and put this line into it.

Doing that will only protect you in that OS X user account, but I bet you only ever use one account on your Mac to SSH anyway.

I updated /private/etc/ssh_config on OS X (system-wide vs. user-specific).

I believe that is the config for the daemon, not the client. I think you want ssh_config.

Yeah, typo'd it in my comment above. Thanks, fixed.

You can patch it with:

  # echo 'UseRoaming no' >> /etc/ssh/ssh_config

  $ echo "UseRoaming no" >> ~/.ssh/config

As mentioned in another thread, don't do this blindly. If there are `Host` or`Match` blocks in the config, this new line will only apply to the last of those blocks.

Thanks DrRobinson ! I added it to my .ssh/config

The second one. There is no /etc/ssh on a stock Mac. (Although there is a /private/etc/ssh_config on mine, without the intervening ssh dir)

Hue? There is one on mine. Maybe mine's different because I upgraded from 10.9 to 10.10 to 10.11 whereas you might have started with a later version (or an earlier one)?

You have an /etc/ssh, or just an /etc? I have /etc, which is a symlink to /private/etc, but as I said it has no ssh dir.

FWIW I am on 10.10.5 not 10.11. Maybe there's a change in 10.11. (The article the grandparent links is focused on 10.10 so I assumed that was in the scope of this discussion.)

PS are you on homebrew? maybe homebrew adds it.

Homebrew should not add anything outside of /usr/local iirc.

Yes, that should be enough, be wary of $PATH issues tho.

It will affect any OSX that is new enough to have that code in it. Same is true with other OS. For instance, on an older, patched FreeBSD 7.2 system, we see this result:

  /root/.ssh/config: line 1: Bad configuration option: UseRoaming
  /root/.ssh/config: terminating, 1 bad configuration option

... which means that sshd predates the roaming code. I haven't tested, but I'll bet my snow leopard workstation also predates that code.

So ... if that line:

  UseRoaming no

produces no errors when you ssh as that user, then you had the problem and you fixed it. If it produces the error above, you never had the problem in the first place (although with an older sshd like that, you should make sure you're not exposed to other, older vulnerabilities).

Ok, an update - snow leopard is indeed too old for this to be relevant. SL with the last and latest software updates still doesn't have the roaming code in sshd and therefore you don't need to fix anything (at least wrt roaming...)

Thanks !

According to Theo, yes.

s/X//

Deleted comment

This does not affect the sshd.

The most terrible thing here is that OpenBSD usually does some full disclosure. This is sure going to be a nasty bug.

Why is this terrible? In any case, the software is open source so any security fix commit can also be considered as full disclosure.

But they didn't disclose the bug yet. The implication is that this bug is /so bad/ that it justified breaking the normal "just disclose it" approach.

http://ftp.openbsd.org/pub/OpenBSD/patches/5.8/common/010_ss...

Leaking private keys seems to be all the rage these days.

I'm curious how typo and bit squatting would come into play here, and if attacks leveraging them could collect private keys at a dangerously high rate before people can patch their clients.

Products like heroku, or the stripe CTF, or other things that come to mind that operate over SSH going rogue a bit scarier. If one were to be compromised it would be a case where mass amounts of private keys could leak. AWS, github, all cloud VPS providers, etc.

Multifactor is relevant as a defense with a vulnerability like this.

This wasn't the only feature that was activated without much documentation. MaxSessions has a significant impact on any systems using 2FA (as the default allows bypassing it if you are phished).

Just like unrestricted sudo, people have because accustom to the bad default behavior of MaxSessions being 10 and allowing un-authenticated multiplexing. (meaning, you auth once, and my trojan can use your session without authentication)

For the single other ChromeOS user (how many of us are there?), note your SSH client is also based on OpenSSH. Add '-o UseRoaming=no' to your arguments.

Watch this repo for updates (or submit a PR?) https://chromium.googlesource.com/chromiumos/platform/assets...

This is a surprisingly bad fail. I've never seen or heard of this option before. In fact, my vimlint for ssh_config doesn't even show it as a valid option!

Check me on this, so this is a client-side problem only, so ssh-ing into only know servers shouldn't be an issue and clients cannot cause problems for servers?

If an enemy takes control of just one of the hosts you ssh into, he will get your private key and can use it to ssh into any other box where you use RSAAuthentication.

I already added the mitigation, but I wondering if my servers can be patched on the weekend.

There is nothing to patch on the server side. You need to ensure the ssh client is updated on the machines you are sshing from.

ok thanks, that's what I figured, but it I was getting a bit worried that I was missing something.

Some good news was added to the updates:

> The agent will never send a private key over its request channel. Instead, operations that require a private key will be performed by the agent, and the result will be returned to the requester. This way, private keys are not exposed to clients using the agent.

ETA: I was way off... you can ignore this... :-)

cf. section 3.3.5 [0], which describes "Roaming (Suspend/Resume)". This is documentation for an application by a company called AppGate (later acquired by Cryptzone) that wrote {some|most} of the code in OpenSSH's "roaming_client.c".

This gives a hint of what the ramifications may be: basically, a MITM, who observed the initial session negotiation, can disconnect the client and hijack an active session.

> Roaming is a feature which allows clients to suspend the connection to the AppGate server and later to resume it again. The user does not need to re-authenticate when reconnecting. Indeed, the entire process can be completely automated and nearly invisible to the user. All established connections will remain alive while roaming. This feature is intended for mobile users who move around between networks.

> Technically, roaming is accomplished by closing the TCP tunnel when the connection is suspended. When resuming, a new TCP connection is made to the server and the SSH data stream is continued through this new connection. The user does not need to authenticate again, instead the client authenticates to the server, without user interaction, with the help of a random password which was made up when the user authenticated at the start of the session. In addition to knowing this password, the client must also know the encryption keys and encryption state to be able to reconnect. It is therefore impossible for a third party to break in and take over a suspended session.

I particularly like that last sentence.

[0]: http://download.cryptzone.com/files/download/AppGate-10.2.3/...

That's not correct though, MITM cannot exploit this. A malicious SSH server can exploit this during a connection though, but only after host key verification.

Oh Cryptzone, funny they offered me a job a few years back. Interesting that this was left in OpenSSH like this, even more as a default.

Could we say it partially because of mono-culture? Almost no one uses GNU lsh as a client/server, dropbear is mostly used as a embedded system sshd. OpenSSH has to many thing integrated and enabled by default.

CentOS/RHEL 6.7 has:

  $ ssh -V
  OpenSSH_5.3p1 OpenSSL 1.0.1e-fips 11 Feb 2013

So... not vulnerable? Posted article says: This affects OpenSSH versions 5.4 through 7.1.

Does this affect mosh?

Yes; mosh depends on ssh to set up the connection (auth and initial key exchange).

No, the roaming in Mosh is unrelated to OpenSSH's roaming feature.

But authentication in Mosh still relies on OpenSSH, so auth could be intercepted by an attacker after which Mosh is completely open to them:

https://mosh.mit.edu/

> However, in typical usage, Mosh relies on SSH to exchange keys at the beginning of a session, so Mosh will inherit the weaknesses of SSH—at least insofar as they affect the brief SSH session that is used to set up a long-running Mosh session.

But Mosh uses OpenSSH for authentication. And if roaming is enabled by default on the client this could still be an issue, no? I'd say it's still a good idea to add "UseRoaming no" to ssh_config.

Right, but mosh uses ssh for initial key setup. I suspect it is possible to exploit that. (As an occasional Windows user who's using the Mosh Chrome extension, I have no idea how much risk I'm at; can't edit ssh_config, it's not even exposed)

Irrelevant; it still affects mosh as mosh does an ssh connection first to do auth and key exchange. That ssh session is potentially vulnerable when connecting to a malicious or compromised host.

Doesn't mosh initially establish a connection via SSH, though?

Is there an explanation of why the roaming feature was removed from the server? It sounds like a useful feature to me and I run into this all the time.

For shells where echo is a builtin that does not know about -e:

printf 'Host *\nUseRoaming no\n' >> /etc/ssh/ssh_config

Am I correct that YubiKey users are unaffected by this vulnerability? As I understand it, the private key never enters into memory.

Using any sort of hardware security module, such as a smartcard (which some YubiKey devices can operate as) will not allow your private key to be compromised because you have no access to your private key -- it's on a whole separate computer (the HSM) and your only access to it is by sending APDUs to that computer. For a typical smartcard the only thing you can ask it to do with your private key is to perform cryptographic operations on data with it. Many smartcard applets require that you authenticate yourself to the smartcard before they will even do that.

I use a smartcard with a NIST SP 800-73 applet on it for all my SSH (and TLS) sessions and do not have any other SSH keys.

I guess it depends on what you mean? I have yubikey 2FA on my servers for SSH, but also have SSH keys. I use SSH keys when I can, and yubikey when I don't have my SSH keys handy.

I suppose if you're doing yubikey+password auth and don't have any keys configured for your ssh client you're fine because... you don't have any keys? :)

You can use a Yubikey NEO to handle key authentication on your computer's behalf:

https://blog.habets.se/2013/02/GPG-and-SSH-with-Yubikey-NEO

Sure, but this capability isn't unique to Yubikey. The question was just worded confusingly. Really it's just "ssh key on a smart card".

Yes, my bad on the wording of the original question.

yubikeys support ssh key auth (via gpg-agent in ssh mode). It's sometimes a bit wobbly, but in general works fine.

This is what I was referring to, and what I'm using, not 2FA. In this case, the PK stored on my YubiKey Neo never makes it into memory.

Does it affect RHEL 6, using openssh 5.3?

Is there an easy way to verify that the workaround is effective? ssh -v doesn't display anything related.

ssh SOMEHOST -v 2>&1 1>/dev/null | grep -i roaming

A vulnerable client will show:

debug1: Roaming not allowed by server

So how does the exploit actually work? How can a malicious sshd actually use this to acquire the private key?

Yet another direct-link post to a cryptic CVE for those of us who aren't in the know on security issues. Upvoting is a great shibboleth to show that some posters "get it", while the rest of us are left in the dark on severity, left to wade through the comments section to try to get a sense of what the real impact is.

From what I can tell, if I accidentally SSH to the wrong server (or a compromised one), my private key can be obtained. I have no clue if that's actually the correct interpretation.

Oh, come on. The issue is very clear. Your interpretation is correct.

Note: this actually means that everyone should regenerate all their key-pairs after updating.

Patch for OS X: echo -e 'Host *\nUseRoaming no' >> ~/.ssh/config

What was this experimental / undocumented roaming feature even suppose to do?

Complete guess: handle changes of ip address on the client side.

Allow a mobile user to reconnect to a disconnected session without (user) re-authentication.

After edit config file , Do we need to rotated/genrate ssh keys also?

After adding in config, Do we need to rota/generate ssh keys also ?

So has anyone run git-blame yet? Who did this?

theo must be mad.

Shaming people for leaving useless non essential feature in their code that results in security breach.

And now the jewel of his crown has been compromised.

The funniest part is now that his jewel has been tarnished, maybe people will understand what he was saying.

And maybe too, people that believed privacy can be achieved on the internet will finally look at the problem of believing the 2 general paradox can be solved without at least 2 different constant link on different plans. And the problem is belief is a poor substitute for thinking - critical thinking.

And maybe people will discover the sad truth of the internet.

Security requires a perfect world, where human beings neither makes mistakes nor are corruptible.

Errare humanum est, perseverare diabolicum

Oh! Some says that is what 2 factor authentication is.

I will answer, my intuition is telling me that 2 factor is good for a fixed amount of time/information and that using it correctly would annoy people to the utmost points.

Then people would say well let's accept that fraud exists. Business first. (costs/benefits)

Then I say giving 3% of all e-commerce to the bad guys is like admitting organized crime have a strong budget for even more crime ... and that we are fucked.

Unless you don't understand that ISIS is basically a startup. A startup that overthrow a state to make even more money and industrialize crime.

We detached this subthread from https://news.ycombinator.com/item?id=10901618 and marked it off-topic.

What Theo (and other OpenBSD developers) have been saying all these years is that it's impossible to /not/ make mistakes, which is why sane design and exploit mitigations are important.

Mad ? Maybe, I wouldn't know. On the other hand, I bet he's really glad all that effort to have ASLR by default was made, because it makes it more difficult for an attacker to exploit vulnerabilites such as this one.

ASLR makes exploits more difficult as long as you have true randomness. It is just a mitigation ... for another problem.

In this case the elephant in the room is stack injection and dynamic libraries. If processes where confined to a well known address space that was self contained ASLR would be useless.. But dependency management would be hellish.

Back to the case. OpenSSH has been openly criticize by 9plan teams for exactly the same reason openSSL has been criticized by openSSH team : too much complexity (not to say plan9 came to anything usable and were right (2 wrongs do not make a right)).

http://harmful.cat-v.org/software/ssh

I do have a feeling as a physicist that the S in CS stands for Shortness of thinking. What is obscure is not profound.

And, at the opposite of engineers I don't believe that security is achievable at all on computers. It is like believing there exists a way to avoid triangulation with one strong radio source.

Computers leak to much information (especially in the physical world), and C is a map that is taken too much for the territory. Every bugs are exploiting a wrong mapping of concepts to implementation.

Modern security is like string theory, admired by everyone because it requires great technical knowledge to master, understood by none because it is way to complex for our "human" brains. We live in an era of belief in solutions.

A bit of critical thinking and of distrusts of experts and stuff that you cannot understand without devoting your life to a subject is at my opinion a must.

I distrust mathematicians for their capacity of dealing with the real world and its uncertainties, and cryptography (as much as functional programming, big data, algorithmic, IA, machine learning) is math driven in a pure Aristotelian world. Where perfection and harmony is the pillar of thinking.

I come from micro-electronics, I only see wires (that are antennas), oscillators, multiplexers, gates and basically a dumb automat I can automatize in respect to time of propagation of signal, and I know that modern computers under the hood are in the physical realm of relativity with approximate answers.

I theorize, build, measure, and retheorize ad nauseum until the product is measured to work the way I expect it to in the domain of validity with a good enough confidence and margins of errors are always on my mind to be controled.

Security requires a zero margin of ambiguity. Physical world is bound to heisenberg equation and coupling. Purity does not exist and this cannot be mitigated. The postulate of cryptography are wrong from the core. Real world always win at the end.

Aristotle way of thinking must die. Math is not science.

What in the hell are you on about?

The base of science : critical thinking as opposed to expertise and "commonly accepted wisdom".

Trust, but check.

And if cannot check, I cannot trust.

Their map (model) is very nice, very detailed and self consistent. But is is not the territory (implementation) and the more complexity we stack the greater we prove the map diverge from the territory. And also the less it can be audited.

Don't expect normal people to trust what they cannot check. It is faith security experts are expecting from users, not trust.

I do my part of the contract as stated by common accepted risk management "best practice" regarding computer security.

I do not trust blindly.

If it's not obvious, don't just add that config option, you have to also restart it.

Actually wait, is this only affecting the ssh client and not the server/daemon ?

Yes, it's client-side only. Server-side, it's disabled (cf. readconf.c).

We went to other way and kissed SSH goodbye a long time ago in favor of immutable infrastructure and automation: https://boxfuse.com/blog/no-ssh

you've never had to debug anything in production, have you?

It's not 2006 anymore. For the vast majority of the cases there is no need for SSH when it comes to logging, profiling and debugging.

Just a note: the default AWS Linux AMI doesn't seem to have this problem on the server side. Connecting to one of my EC2 instances with verbose on I get the following message:

  debug1: Roaming not allowed by server

Yeah! AWS Linux for the win. :)

This is client-only vulnerability:

   The matching server code has never been shipped, but the client
   code was enabled by default and could be tricked by a malicious
   server into leaking client memory to the server, including private
   client user keys.[1]

[1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-Ja...

That message actually means that your client has roaming enabled and is therefore vulnerable.

OpenSSH server doesn't support roaming. This is a client only issue. The problem is that your connection could be MITM'd by someone looking to exploit this bug.

> your connection could be MITM'd

MITM isn't a risk, if I understand this statement in the undeadly.org announcement:

   The authentication of the server host key prevents exploitation
   by a man-in-the-middle, so this information leak is restricted
   to connections to malicious or compromised servers.

Unless it's your first connection to a legit uncompromised server, yes? (AWS instance, etc)

Reply on news.ycombinator.com