Comment by kelnos | Hacker News Reader

Comment by kelnos | Hacker News Reader

Parent Full thread

Wow, your assessment of the impact here (or even possible impact) is way way way overblown.

In reality, the vast majority of users don't pay attention to SSH host keys at all.

Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections of their target.

Next, they have to decide what they want to do.

If they want to send malicious code to someone doing a 'git pull', they'd have to craft the payload specifically for the repo being pulled from. Not impossible, but difficult.

If they want to "steal" source code from someone doing 'git push' (perhaps to a private repo on GitHub), that's a bit easier, as they can just tell the client "I have no objects yet", and then the client will send the entire repo.

And, again, they'd have to have the ability to MitM some git user's connections. Regardless, there is no way that they could change any code on github.com; this key would not give them any access to GH's services that they don't already have.

So I think your anger here is a bit over the top and unnecessary.

I agree that it's pretty bad that some GH employee even had the ability to get hold of this private key (sure, probably should be in an HSM, but I'm not particularly surprised it's not) in order to accidentally leak it, but... shit happens. They admitted the mistake and rotated the key, even though it's likely that there was zero impact from all this.

End users do not pay attention but their clients pin the key after first use. Also everyone is using gitops these days and almost no one is using dns-over-tls.

Imagine you control the right router on a company wifi, or any home wifi a production engineer works from and suddenly you can cause them to clone the wrong git submodule, the wrong go package, or the wrong terraform config.

If you knew a CI/CD system blindly clones and deploys git repos to prod without signature checks, and that prod is a top 10 crypto exchange with 1b of liquidity in hot wallets, then suddenly a BGP attack to redirect DNS is a good investment. Myetherwallet got taken over for 15 minutes with a BGP so this is not hypothetical.

Should that be the case? Of course not. But the reality is I find this in almost all of the security audits I do for fintech companies. Blind trust in Github host keys is industry standard all the way to prod.

Sure I can imagine that. And in doing so, I imagine this attack is pretty unlikely.

I mean, think of the confluence of things that have to line up for this to work for someone. Many stars have to align in order for someone to successfully exploit this leak. Is it impossible? No, of course not, and so GH was right to rotate the key here, even if their server request logs suggested that no one had accessed it.

If people actually have this sort of attack in their threat model, there are ways to protect against it. Signing commits and verifying them. Pinning to a particular git SHA1 hash. Etc. If people are not doing these things, then it's possible they've made the decision not to worry about this sort of attack. Sure, you can disagree with, but I think you'd probably be in the minority. That's ok, though; you can certainly protect the things you're responsible for in stronger ways.

I have seen attacks along the lines I just outlined and well beyond in my industry many times.

A wildcard TLS cert or an SSH host key in plaintext is a loaded weapon laying around and it will be used on a high value target.

Sad fact is most of the companies that hold billions of dollars of customer assets do development exactly the same way a gaming company might. Those that even attempt security are unicorns. They bank everything on things like DNS, BGP, TLS certs, and ssh host keys working. This is like the medical industry before they learned hand washing was a thing.

I teach every one of my clients how to do things like commit signing, but for every one I help improve there are 100 more on track to be hacked any day now.

I can totally forgive a startup that cannot afford senior security engineers for a mistake like this, but Microsoft can afford that, or at least consultants, and yet they can not even enable optional signing for NPM, properly enforce git commit signing, or even protect an ssh host key trusted by millions in a TEE or HSM.

Yes, but they will have to un-pin the (now compromised) key if they want to continue using Github.

Any compromise would have to isolate them from the "real" Github hosts from today onwards, i.e. plausibly MITM them continuously, or they would just switch to the rotated key to be able to continue working. At least in OpenSSH, this means replacing the compromised trusted key, as there can only be one per host (or even IP in the default configuration).

This is still very bad, but much less catastrophic than e.g. a world in which `.ssh/known_hosts` allows multiple entries, in which case you'd really have a sustained compromise of most clients.

Woah, I looked around some more and it seems like the opposite is true. Multiple trusted keys per domain can exist, and additionally there is this option:

> UpdateHostKeys is enabled by default if the user has not overridden the default UserKnownHostsFile setting and has not enabled VerifyHostKeyDNS, otherwise UpdateHostKeys will be set to no.

This is an OpenSSH extension that allows any host to provide additional host keys the client will then silently add to `known_hosts`. This is really bad in this context as it can allow a one-time MITM to install rogue keys for github.com that can go unnoticed; check your `known_hosts` for any unexpected entries for github.com!

Crypto never fails to impress me with how many problems it creates for itself.

> Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections

This is not hard. If it were hard, we wouldn't need encrypted connections.

If I were a nation state, it would be trivial to position an attack at the edge of a network peering point for a given service. How do I know? Our own government did it for 10+ years.

Cyber criminals often have the same tactics and skills and can find significant monetary reasons to assume heightened risk in order to pull off a compromise.

Random black hats enjoy the challenge of compromising high value targets and can find numerous ways to creatively infiltrate networks to perform additional attacks.

Even without gaining access to a network, virtually anyone on the internet can simply push a bad BGP config and capture traffic for an arbitrary target. Weird routing routinely happens that nobody can definitely say isn't such an attack.

Reply on news.ycombinator.com