Hacker News Reader

Comment by p-e-w | Hacker News Reader

It never fails to amaze me how most incident mitigations seem completely oblivious to such security side effects.

"We have no reason to believe that the exposed key was abused, but out of an abundance of caution, we are going to expose 50 million users to a potential MITM attack unless they are extremely careful."

Not a single word in the post about whether this impact was even considered when making the decision to update the key. Just swap the key, and fuck the consequences. Same with the mass password resets after a compromise that some services have done in the past years. Each of those is any phishing operation's dream come true.

View on HN

Don't trust corporate PR. They're obviously lying when they say "out of an abundance of caution". The private key was exposed in a public GitHub repo, it could literally be anywhere.

So MITM for some of 50m users is strictly better than MITM for all of 50m users.

It could be, but also GH might be logging inbound requests long enough to see whether the file was requested.

> The private key was exposed in a public GitHub repo.

How do you know this?

Github runs scanner for private keys in public and private repos and notifies owner (I did it once so I know ... ;)). So some Github engineer likely would have received such an email if what you say is true. Hilarious.

From the OP:

> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository.

It says exactly that in the article:

> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository

Hilarious.

I didn't read it fully before commenting. I'm sorry.

If everyone read the entire article of each HN submission the comments section would be wildly different :-)

I can’t see that could ever happen. Is the key just floating around on their employees computers and skeins accidentally committed it?

Maybe because the article says so?

> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository

After reading first paragraph, I was sure they don't have any specific reason to replace it.

> leaked in public repo

Me: Yeah, that's why they are doing it.

I'm always amazed at this kind of posts. Did these 50 million users (surely none of them use git+https!) check the host key the first time they connected to github? Did you?

It doesn't matter because it didn't change! That's the beauty of TOFU.

The point is, what if it you were MITMed from the beginning?

Sure, but the difference is that it's now both a plausible moment to go MITM (because they got that key), and furthermore the hypothetical attacker now has good reason to believe users won't be scared by a host-key-change warning, and the hypothetical attacker would know this opportunity exists for a large set of users simultaneously. If some malicious network operator were to try and exploit users, now would be a good moment - they'd likely catch many more people in the time it takes to be discovered than on an average day.

The MITM-at-the-start risk is of course real, but I think this new everyone -restarts-simultaneously risk is qualitatively different enough to be worth at least considering.

Much more concerningly, there is an activated-by-default OpenSSH extension (`UpdateHostKeys`) that allows the server to install new host keys into `.ssh/known_hosts` after every successful server authentication.

The bad guys would also have to have MITMed it every time I connected for the last 15 years, or I would have seen authentication failures when it connected to the real thing. MITMing someone once isn't that hard, but doing it consistently is.

It doesn't matter because it didn't change! That's the beauty of TOFU.

One way to solve this in TOFU is to have a time window where both keys are presented.

If we're starting from the assumption that the first key was compromised, then you're still vulnerable to MITM. The only solution is communicating the key through a different, trusted way. Which is exactly what github did - inasmuch you can trust that github.com is github.

> (surely none of them use git+https!)

well, yes

github doesn't accept https push any more

Not sure what you mean. You can push via HTTPS: https://docs.github.com/en/get-started/getting-started-with-...

Maybe you’re referring to how they no longer accept passwords for HTTPS auth? You have to auth for HTTPS push with a personal access token.

That's funny, I do it every day. It's frankly easier to install git credential manager (even integrate into WSL) for 2FA authentication on Github (and other git hosts).

I get a bit paranoid when having to deal with Tokens on various CI/CD environments as it stands. And the things that start breaking every year when I forget to update them. Note: this is personal/hobby projects, not corporate stuff, where I'm strictly in the codebase and try to keep my fingers out of CI/CD beyond getting a Docker image built, and someone else configures the keys/auth.

How are you using git credential manager for 2fa on GitHub? They stopped supporting user/password auth for HTTPS git access a while back, and started requiring personal access tokens (which do not require a 2nd factor)

GCM will use an embedded browser so you can authenticate with the UI including your second factor, which will then give you a credential/token that can be used in the git environment over HTTPS. It's still a (differt, oath vs reference generation) token, but you aren't having to go generate, configure and update it yourself.

> Did you?

I sure did. Doesn’t everyone?

The point being made here is that because there are millions of users forced to change keys now all at around the same time, and because they are doing so due to seeing an error in Git, this creates a good opportunity to strike. Normally, most users would have connected to GitHub before and a MITM attack has a high chance of failure.

While their reaction is more likely to cause a security breach, consider the psychology.

If the key was breached and Github just didn't know it, then a breach happened, then only Github would be to blame.

If Github rotates its key, and somebody suffers a MITM attack, the blame is more diffuse. Why didn't they verify the key out of band?

How is there an alternative here?

There isn't an alternative, really. The private key has been exposed, and presumably it is unknown if or how far it has spread. The SSH keys must be changed, and the sooner the better. All that can be done is to notify people after the change has occurred.

> and presumably it is unknown if or how far it has spread

Why would that be unknown? GitHub has HTTP and SSH access logs, right?

Since the post doesn't mention anything like "we reviewed logs and confirm the key was not accessed", it is very likely that they either don't have logs that are reliable enough to rule it out (e.g. they may be sampled or otherwise incomplete), or that the key was accessed.

Keeping a complete log of all GET requests to random files in a public repository in a reliable way would be insane.

No, it wouldn’t be - assuming by “insane” you mean “silly to do”. I build systems at Google that do exactly that.

Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.

Presumably, keeping 'last remotely accessed' and 'last remotely modified' for every file (or other stats that are a digest of the logs) is sane for pretty much any system too. Having a handle on how much space one is dedicating to files that are never viewed and or never updated seems like something web companies that have public file access would all want?

What guarantees do these systems provide? Are 100% of requests where data was served guaranteed to either end up in the log or at least create a detectable "logs may have been lost here" event?

Or does it log all the requests all the time as long as everything goes well, but if the wrong server crashes at the wrong time, a couple hundred requests may get lost because in the end who cares?

It's not just GET requests. Someone could have cloned/refreshed the repo using ssh. The repo might have been indexed by github's internal search daemon which might not use the public HTTP API but uses internal access ways however those might look like. You might have purged the database of that daemon but what about backups of it? What about people who have subscribed to public events happening in the github.com/github org via the API?

You'd have to have logging set up for all of these services and it would have to work over your entire CDN... and what if a CDN node crashed before it was able to submit the log entry to the log collector? You'll never know.

One H4X0R gave it to four friends, who in turn gave it to between 9 and 14 friends, who in turn gave it to between one and 6 friends.

If train A leaves New York going 60 miles per hour with 167 people on board and train B leaves Chicago one hour later with 361 people on board going 85 miles per hour, how many people now have the key?

The answer is 31337.

Same as with any other decision: Do a cost/benefit analysis of whether the security risk created by rotating the key is actually outweighed by the security risk of doing nothing, taking into account logs that should tell you whether the exposed key was indeed accessed by unauthorized parties.

To be 100% clear: Both courses of action come with associated security risks. The problem is not choosing one course of action over the other, the problem is thinking you can just skip the cost/benefit analysis because the answer is somehow 'obvious'. It's not obvious at all.

No, you cannot keep using an exposed key. You must replace it. There is no cost/benefit analysis needed in this situation.

Wrong. A CBA is always needed. If the potential damage from MITM attacks made possible by rotating the key is greater than the potential damage from a rogue key multiplied by the likelihood that someone actually accessed the key, then it is wrong to rotate the key. It's that simple.

The only way a CBA would be unnecessary is if rotating the key didn't have any security risks. But it does.

Here I’ll do the CBA:

- if they have evidence that the key was exposed to one person, even with zero usage of the key, failing to rotate the key is tantamount to knowingly accepting widespread compromise at a potential attacker’s whim. At GitHub’s scale, that’s untenable.

- rotating the key is the only correct reaction to that

- they should have better communications in place to help users mitigate MITM

- there really isn’t an option, because they’re critical infrastructure; I’m glad they know that and acted accordingly

- on principle this speculation makes sense, but understanding the threat makes it moot

- you hopefully know that, and it’s good to insist on thoughtful security practices but it’s important to also understand the actual risk

There is a MITM risk regardless of whether they rotate the key. Except one is a one time risk and the other is a perpetual risk.

Thus rotating is the only logical course of action.

Only if you know for certain that the key has been accessed by a third party.

If you don't know for certain, you have to factor in the likelihood that it has been, and at that point, the two risks aren't equal anymore so that logic doesn't work.

What if you don't know for certain ?

You just ignore it and hope for the best ?

Only if you are certain (and better be really sure you haven't missed any cache/cdn, temp files backus etc.) it wasn't accessed you do nothing.

It was publicly exposed, and if they are making this announcement it’s essentially guaranteed they can’t rule out it was accessed.

Are you arguing for the sake of arguing and technical correctness or do you actually believe Github shouldn't rotate their key in this situation?

What? This is a terrible way to reason about risks in general. If you don't know for certain, you should assume the worst case scenario, especially since it's impossible for you to calculate the probability distribution of the likelihood of a leak.

You should only keep moving along without key rotation if you know for 100% certainty a leak didn't happen and no one accessed the key (not theoretically impossible if they had the server logs to back it up), but anything minus that and you have to assume it's stolen.

That's why you need certificates and not just a key pair. Certificates make key rotation easier, and you want key rotation to be easy.

I guess the proper way forward is a small utility that gets the latest signature through http+tls, and replaces the line in your known_hosts file, all in the background.

Looking long term, maybe we need to get rid of all the security stuff in ssh and just pipe the rest of its functionalities inside a TLS pipe. Let the os do its certificate management, reuse security bricks that are way more studied, ...

Certificates just add more keys to worry about. The beauty of SSH is that it does not add hugely trusted parties in the name of convenience, while the UX of TOFU (trust on first use) is pretty decent.

The real solution to break out of these UX/security tradeoffs is to put domain names on a blockchain: then you can simply rotate the key in your DNS record, while the blockchain model is such that you need to compromise many parties, instead of "one out of many parties", as with CAs.

Tracking Bitcoin chain for DNS updates is lightweight enough that it can be built into OS alongside other modern components such as secure enclave, TCP/IP stack and WiFi/BT/5G radios.

> The beauty of SSH is that it does not add hugely trusted parties in the name of convenience

Even with a certificate authority model, you don't have to trust any CAs if you don't want to. Not having the option to do so is more of a problem.

We should use a separate system that could reliably verify which certs belong to which entity.

Blockchain is a perfect solution to this. I wonder why it is not considered yet.

https://certificate.transparency.dev it is

Thanks, I was not aware!

Those keys can be worried about on a better secured computer, and don't need to be spread out on every frontend ssh server. Also it allows you to have each machine have a different host key pair, so if one leaks, only that single machine may have some trust issues, and not the whole fleet.

Also it's way better than TOFU, you can just add the CA key to known_hosts and avoid TOFU for each machine.

(Nevermind that you'll probably not accidentally commit some semi-ephemeral host key that's rotated often somewhere, because it will not be some special snowflake key you care about, but something handled by your infrastructure software automatically for each machine)

Are there any cert solutions that dont involve having to maintain a revocation list? I only used certs with openvpn years ago and the CRL was a potential footgun.

This is one reason people are issuing certs with 2 week expiry.

> Certificates make key rotation easier

How easy is it to rotate the keys of your CA?

The alternative would be to use certificate authorities (ssh has CA support) which allow to effectively have private keys at different levels and allow you to keep the root private key in a physical vault and use it very rarely to issue other private keys

And then don't forget to setup key revocation as well, and make sure that an attacker in a position to MITM the connection cannot cause the revocation checks to fail-open.

I hope you don't need that SSH connection to fix your broken CRL endpoint!

This would just offload the problem to a separate entity. CAs can be (and have been) compromised.

Sure, but isn't it more likely that a key that has to be shared by who knows how many ssh load balancer machines at GitHub and can't be easily rotated because it's pinned by millions of users, isn't it more likely that that private key gets eventually compromised or thought to be at risk at being compromised?

We need to compare the relative risks within the same context, namely within a company like GitHub

So it's not relevant to bring up failures of other CAs

Jump into a time machine, go back to the creation of SSH, and adopt SSL-style trusted third-party certificate authorities. Somehow get it adopted anyway, even though loads of people use SSH on internal networks where host-based authentication is difficult; SSH is how many headless machines are bootstrapped; and that you've got to do it 19 years before Lets Encrypt.

Jump into a lesser time machine, go back to when Github were creating their SSH key, and put it into a hardware security module. Somehow share that hardware-backed security key to loads of servers over a network, without letting hackers do the same thing. Somehow get an HSM that isn't a closed-source black box from a huge defence contractor riddled with foreign spies. Somehow avoid vendor lock-in or the HSM becoming obsolete. Somehow do this when you're a scrappy startup with barely any free time.

Ssh certificate authorities are a thing that exists.

We also have a way to put SSH host key fingerprints in DNS records already.

Yes but the option to do verify host keys using ("VerifyHostKeyDNS") is not enabled by default.

DNS can trivially be mitm'd. DNS-stored fingerprints are strictly less secure than TOFU.

If you use DNSSEC (cue inevitable rant from Thomas) this just works. If you have DoH (and why wouldn't you?) and your trusted resolver uses DNSSEC (which popular ones do), you get the same benefits.

https://en.wikipedia.org/wiki/SSHFP_record

Yeah like how HTTPS CAs exist. There are some very nice three letter ones who can issue any certificate and your browser / OS happily accepts it.

SSH doesn't have any CAs that it trusts out of the box. It's up to you to tell it which one to trust.

Unless it has changed recently, you can't have a trust chain of OpenSSH certs though so it's cumbersome that your signing key is not only the root ca but also basically has to be 24/7 accessible to sign any server/client you want to bring up.

This just kicks the can down the road to DNS.

I'd guess that most systems aren't using DoH/DoT or end-to-end DNSSEC yet. Some browsers do, but that doesn't help tooling frequently used on the command line.

I suppose you could just accept X.509 certificates for some large/enterprise git domains, but that pokes up the hornet's nest that is CA auditing (the browser vendors are having a lot of fun with that, I'm happy that the OpenSSH devs don't have to, yet).

And where do you maintain the list that decides which hosts get to use TOFU and which ones are allowed to provide public keys? Another question very ill-fitted for the OpenSSH dev team.

No browser uses DNSSEC.

That was in reference to the former, i.e. in-browser DoH/DoT lookups.

Thank god. Someone needs to take that protocol out back and give it the old yeller treatment.

There could be an update to the protocol that enables certified keys to be used and allows them to be accepted without warning or with less of a warning.

There could be a well known URL that enables people to fetch ssh keys automatically in a safe manner.

> and adopt SSL-style trusted third-party certificate authorities

So that any large entity can own your servers with easy. (Well, they already can, but not through this vulnerability.)

Anyway, the only thing CAs do is to move that prompt into another, earlier time. It's the same prompt, the same possibility for MITM, and the same amount of shared trust to get wrong. You just add a 3rd party that you have to trust.

SSH does have a CA system. Anybody that isn't managing a large datacenter will avoid it, for good reason.

> So that any large entity can own your servers with easy.

Eh, let's not pretend existing SSL certificate validation is anything to write home about.

Even without any ephemeral servers involved, barely anybody is validating cert fingerprints on first use.

And among people using ephemeral servers, 99% of applications have either baked a certificate into their image (so that any compromised host means a compromise of the critical, impossible-to-revoke-or-rotate key) - or every new server gets a new cert and users have either been trained to ignore certificate warnings, or they've disabled strict host key checking and their known hosts file.

The existing SSL cert validation options are perfect if you're a home gamer or you're running a few dozen bare metal servers with all your SSL users within yelling distance in the same office. But we all know it's a joke beyond that.

Clone repos using oauth2 with two factor enabled - both GitHub and GitLab support that though their CLIs.

Well, at least SSH could allow for signing a new key with the old one. So they could say it's signed, and people would know to accept only a different prompt.

There is DNS verification, but people have been trained all their lives to accept insecure DNS information (and set their systems accordingly), and I really doubt the SSH client checks the DNSSEC data.

How would one stage a MITM attack without knowing the private key corresponding to the old key?

They don’t need it. Millions of users are going to blindly trust the “new” GitHub public SSH key they see the next time they connect without checking to see if it matches the published signature.

By pretending to be the host that the user is trying to connect to. You can then present the client with a key you generated yourself. Of course, SSH will warn the user that the fingerprint has changed, but they'll just think "Ah yes, GitHub changed their keys so it's probably fine." This is why updating the key creates a potential MITM risk, unless people actually bother to verify that the fingerprint is correct.

What specifically can you verify that cannot also be spoofed? If I go do this now I (and I’m sure millions of others) have no idea what to look for. I’d be blindly accepting like a sheep if it weren’t for this thread.

edit: I found this helpful and honestly had no idea I should be doing this (I’m a hobbiest not a professional) https://bitlaunch.io/blog/how-to-check-your-ssh-key-fingerpr...

if you run a MITM attack today the victim gets the warning in the blog post. Thier most likly action is to google the blog post and see it is expected and and so accept your fake key. Having said that I dont see what choice Github had, they can't continue to use a leeked key.

Their solution is the second thing in the blog post which is demand into your known host file.

The problem here is that their first command that they advise using is the one that removes the old key and most users are just going to stop right there because it solves the problem of getting the key warning.

The right solution here is to provide a command that most users are going to copy and paste that deletes the old key and adds the new key all at once.

The fact that users have to delete the old Github key from their systems and accept a new one is what could lead to a MITM attack.

If your system doesn't know the public key of an SSH server, when you connect the first time, the SSH client will display a warning and ask you if you accept the server key. An attacker could be between you and Github and if you accept without checking it's the correct key, you would be toast.

Would it be more secure to access a https secured server to get the keyfile then?

Yes, GitHub's announcement provides the correct new public RSA key, and it also provides instructions for a curl invocation which does all the work if you don't trust yourself to copy-paste text or don't understand how.

Only if the https server cert wasn't compromised at the same time as the ssh key. For all we know, this entire announcement of "we have a new key" could be staged.

The key has changed, meaning that every user has to accept a new key.

Meaning that a lot of users will blindly accept whatever new key (even when it might be the one owened by attacker doing MITM) because Github, their college or random person on internet said that that's what you have to do to get rid of error.

Fortunately this will become evident once they connect from elsewhere and the key changes again

Not strongly evident. I suspect most users would assume they did something wrong, or that GitHub was still making changes.

> Meaning that a lot of users will blindly accept whatever new key (even when it might be the one owened by attacker doing MITM)

This is less likely because unlike for TOFU the SSH client just rejects the mismatch and insists you take manual action, and the likely manual action will be "Paste this stuff from the announcement".

So an adversary needs to either subvert whatever messaging you see (which is tricky, likely impossible for a random user visiting the github web site wondering what's wrong) or hope that you just try to muddle along and do TOFU again, putting you in the same spot as a whole bunch of users every day at GitHub.

Is there a benefit (and practicality) in recording encrypted traffic by an adverse intermediary waiting for keys being exposed sometime? Like now?

No, the risk of losing an SSH host key is less this (because of forward secrecy), rather impersonation of the server.

Reply on news.ycombinator.com