Hacker News Reader

Wow, your assessment of the impact here (or even possible impact) is way way way overblown.

In reality, the vast majority of users don't pay attention to SSH host keys at all.

Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections of their target.

Next, they have to decide what they want to do.

If they want to send malicious code to someone doing a 'git pull', they'd have to craft the payload specifically for the repo being pulled from. Not impossible, but difficult.

If they want to "steal" source code from someone doing 'git push' (perhaps to a private repo on GitHub), that's a bit easier, as they can just tell the client "I have no objects yet", and then the client will send the entire repo.

And, again, they'd have to have the ability to MitM some git user's connections. Regardless, there is no way that they could change any code on github.com; this key would not give them any access to GH's services that they don't already have.

So I think your anger here is a bit over the top and unnecessary.

I agree that it's pretty bad that some GH employee even had the ability to get hold of this private key (sure, probably should be in an HSM, but I'm not particularly surprised it's not) in order to accidentally leak it, but... shit happens. They admitted the mistake and rotated the key, even though it's likely that there was zero impact from all this.

End users do not pay attention but their clients pin the key after first use. Also everyone is using gitops these days and almost no one is using dns-over-tls.

Imagine you control the right router on a company wifi, or any home wifi a production engineer works from and suddenly you can cause them to clone the wrong git submodule, the wrong go package, or the wrong terraform config.

If you knew a CI/CD system blindly clones and deploys git repos to prod without signature checks, and that prod is a top 10 crypto exchange with 1b of liquidity in hot wallets, then suddenly a BGP attack to redirect DNS is a good investment. Myetherwallet got taken over for 15 minutes with a BGP so this is not hypothetical.

Should that be the case? Of course not. But the reality is I find this in almost all of the security audits I do for fintech companies. Blind trust in Github host keys is industry standard all the way to prod.

Sure I can imagine that. And in doing so, I imagine this attack is pretty unlikely.

I mean, think of the confluence of things that have to line up for this to work for someone. Many stars have to align in order for someone to successfully exploit this leak. Is it impossible? No, of course not, and so GH was right to rotate the key here, even if their server request logs suggested that no one had accessed it.

If people actually have this sort of attack in their threat model, there are ways to protect against it. Signing commits and verifying them. Pinning to a particular git SHA1 hash. Etc. If people are not doing these things, then it's possible they've made the decision not to worry about this sort of attack. Sure, you can disagree with, but I think you'd probably be in the minority. That's ok, though; you can certainly protect the things you're responsible for in stronger ways.

I have seen attacks along the lines I just outlined and well beyond in my industry many times.

A wildcard TLS cert or an SSH host key in plaintext is a loaded weapon laying around and it will be used on a high value target.

Sad fact is most of the companies that hold billions of dollars of customer assets do development exactly the same way a gaming company might. Those that even attempt security are unicorns. They bank everything on things like DNS, BGP, TLS certs, and ssh host keys working. This is like the medical industry before they learned hand washing was a thing.

I teach every one of my clients how to do things like commit signing, but for every one I help improve there are 100 more on track to be hacked any day now.

I can totally forgive a startup that cannot afford senior security engineers for a mistake like this, but Microsoft can afford that, or at least consultants, and yet they can not even enable optional signing for NPM, properly enforce git commit signing, or even protect an ssh host key trusted by millions in a TEE or HSM.

Yes, but they will have to un-pin the (now compromised) key if they want to continue using Github.

Any compromise would have to isolate them from the "real" Github hosts from today onwards, i.e. plausibly MITM them continuously, or they would just switch to the rotated key to be able to continue working. At least in OpenSSH, this means replacing the compromised trusted key, as there can only be one per host (or even IP in the default configuration).

This is still very bad, but much less catastrophic than e.g. a world in which `.ssh/known_hosts` allows multiple entries, in which case you'd really have a sustained compromise of most clients.

Woah, I looked around some more and it seems like the opposite is true. Multiple trusted keys per domain can exist, and additionally there is this option:

> UpdateHostKeys is enabled by default if the user has not overridden the default UserKnownHostsFile setting and has not enabled VerifyHostKeyDNS, otherwise UpdateHostKeys will be set to no.

This is an OpenSSH extension that allows any host to provide additional host keys the client will then silently add to `known_hosts`. This is really bad in this context as it can allow a one-time MITM to install rogue keys for github.com that can go unnoticed; check your `known_hosts` for any unexpected entries for github.com!

Crypto never fails to impress me with how many problems it creates for itself.

> Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections

This is not hard. If it were hard, we wouldn't need encrypted connections.

If I were a nation state, it would be trivial to position an attack at the edge of a network peering point for a given service. How do I know? Our own government did it for 10+ years.

Cyber criminals often have the same tactics and skills and can find significant monetary reasons to assume heightened risk in order to pull off a compromise.

Random black hats enjoy the challenge of compromising high value targets and can find numerous ways to creatively infiltrate networks to perform additional attacks.

Even without gaining access to a network, virtually anyone on the internet can simply push a bad BGP config and capture traffic for an arbitrary target. Weird routing routinely happens that nobody can definitely say isn't such an attack.

The key has to be in memory on all of their front end servers. Do you think a quorum of engineers should get together every time a front end server boots or reboots?

Genuinely asking because I’ve struggled with this question.

Lots of cloud instances support remote attestation these days which gives you a reasonable path to autoscaling secure enclaves.

1. You compile a deterministic unikernel appliance-style linux kernel with a bare bones init system

2. You deploy it to a system that supports remote attestation like a nitro enclave.

3. It boots and generates a random ephemeral key

4. m-of-n engineers compile the image themselves, get the same hash, and verify the remote attestation proof confirming the system is running the bit-for-bit trusted image

5. m-of-n engineers encrypt and submit shamirs secret shares of the really important private key that needs protecting

6. key is reconstituted in memory of enclave and can start taking requests

7. Traffic goes up and autoscaling is triggered

8. New system boots with an identical account, role, and boot image to the first manually provisioned enclave

9. First enclave (with hot key) remotely attests the new enclave and obtains its ephemeral key (with help of an internet connected coordinator)

10. First enclave encrypts hot key to new autoscaled enclave

11. rinse/repeat

I don't understand how initializing cryptographic keys from an HSM at boot time is an untenable proposition. The quorum would be for accessing the key by human means. You can have a separate, approved path for pre-authorized machines to access cryptographic primitives across an isolated network.

The key doesn't have to in memory on all of their front end servers. Any respectable company that cares about security wouldn't put their TLS private key on all of their front end servers anyways. You expose a special crypto oracle that your front end servers talk to; the oracle can be a specially hardened process on a dedicated server or better yet a HSM; the point is, the private key is never in memory on any server that handles untrusted data.

> With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right?

This is unfortunately not how SSH works. It needs to be unlocked for every incoming connection.

You raise valid hypotheticals about the security of the service... but fixing it involves removing SSH host key verification from Github; better OpsSec would not fully resolve this issue.

I am well aware how ssh works. I have written ssh servers and design secure enclave key management solutions for a living.

Even if they wanted the most quick and dirty lazy option with no policy management, they could stick the key in a PKCS#11 supporting enclave every cloud provider supports these days. OpenSSH natively supports them today.

At a minimum they could have placed their whole ssh connection termination engine in a immutable read only and remotely attestable system like a Nitro enclave or other TEE. You do not need to invent anything new for this.

There are just no excuses here for a company with their size and resources, because I do this stuff all the time as just one guy.

Would these secure storage methods for the key be able to scale up to the ssh connection volume an outfit like GH sees? Genuinely asking; I don't know the answer. I just feel like having to hit a HSM or something similar every time a new ssh connection comes in wouldn't work. Or at the very least I would not see the sub-100ms connection time I can get right now.

Easily. You could have only the CA key in the enclave and have it sign throw-away session keys on stateless autoscaling machines, and/or you can have your actual ssh terminations happen in cloud enclaves like nitro enclaves which have all the same specs and cost of a regular cloud server.

Hardware security modules can perform key operations without allowing anyone to access the key data. Key material being (accidentally or deliberately) leaked has been a solved problem for a long, long time.

Used in such a way, the entirety of Github's SSH connections would be bottlenecked by this HSM. It wouldn't scale and it would be a single point of failure. As lrvick points out, you'd have to use a certificate-based host key scheme like PKCS#11 to make this scalable. That's fine, but it is a different scheme than RSA host key identification.

I gave two options but they are separate.

PKCS#11 is a protocol for talking to hardware security modules for generic cryptographic operations to keep a key out of system memory.

You can totally take a single host key scheme and use HSMs or TEEs at scale.

Openssh supports using openssl as a backend which in turn can use the PKCS#11 protocol to delegate private key operations to remote enclaves or TEEs. Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else.

> Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else.

Isn't having one key the root of the problem? If you've got one key, it has to be transferred to each HSM module, which means it's going to be in their deploy code. My understanding is that the safe way to scale PKI is to have an HSM generate a key internally, and have that key be signed by a CA in a different HSM, so private key material never leaves any HSM.

I guess you're saying that SSH RSA host keys support this, but I'm only familiar with doing it using looks up correct terminology X.509 certificates like for HTTPS.

> If you've got one key, it has to be transferred to each HSM module, which means it's going to be in their deploy code

There's products out there that allow for keys to be securely shared amongst multiple HSMs, without the key ever existing in clear text outside the HSM.

HSMs can key-wrap a key.

E.g. Foo can encrypt the key X with Bars public key and then Bar can import key X.

Most HSMs have modes that allow for load to be distributed amongst multiple HSMs.

> It needs to be unlocked for every incoming connection.

Yep. Well, certificates exist exactly to bridge the GP's requirement with your reality.

Which HSM are you looking at that would be able to handle the required number of transactions per second?

The same ones that terminate TLS for millions. Most are garbage but at least they keep the key offline. Also you can scale horizontally or only use the HSM as a CA to sign short lived host keys.

You could also use things like Nitro enclaves which have all the same specs as a regular EC2 instance.

Tons of options. They clearly chose none of them.

> The same ones that terminate TLS for millions.

Which ones?

> Also you can scale horizontally

For SSH? Only by having the same private key in all of them, which means it's still around somewhere.

> or only use the HSM as a CA to sign short lived host keys

That would be ideal, except that the user experience around SSH host certificates is currently woeful.

(Not your parent commenter.)

> Which ones?

Take Utimaco Security Server and you'll get 10k+ RSA signatures per seconds. Yes, you'll definitely need dozens of them, but you'll probably want several for high availability and low latency anyway.

> For SSH? Only by having the same private key in all of them, which means it's still around somewhere.

End-to-End encrypted key transfer between HSMs is well established. Ops for such a setup is definitely going to be a pain and lots of manual (and thus expensive) work but it is doable. The banking industry has been operating like that since forever – with symmetric cryptography only. Imagine two people being sent letters with XOR halves of a transport key and physically meeting at an HSM and entering the halves on a PIN pad (not a hexadecimal but a decimal PIN pad, mind you, where you need to press shift for A-F). From a modern perspective it's totally bonkers, but it works.

If I was tasked with building something for large scale companies like GitHub, I would probably pass up on HSMs and use commodity measured boot on a minimal Linux for my trusted key servers. Outside of these key servers the SSH key would only be stored split up with Shamir-Secret-Sharing with few trusted employees which will only restore the key on ephemeral offline systems. Is that overkill? Very much depends on your threat model. Investing in the security of their build chain runners and data-at-rest integrity might have higher pay off. But then again, GitHub has become such a big player that we should also hold them to very high standards. And the setup can be re-used for all their secret management, e.g. TLS certificate private keys.

Thanks! That's hugely helpful - everything I could find myself seemed to be an order of magnitude or so slower than that (and they support ecdsa, too, so I can't even object on the basis of algorithmic support). With hindsight my reply was somewhat flippant - really I just wanted to push back on the idea that this was a problem that could be solved by simply sprinkling HSMs on it rather than something that requires a meaningful infastructural effort. Github is a sufficiently core piece of infrastructure that I agree more should be expected, and I hope this does serve as encouragement for them to do that.

The HSM only needs to sign new host keys, transactions per decade at thier current rate.

Thoughts with the sysadmins and devops people out there on this wonderful Friday afternoon.

These kinds of changes suuuuuuck. Messing with known_hosts file is not always something easy to do. Might require a image rebuild, if you have access at all.

Is it negligence or just incompetence? I get the sense that security is such a tough problem that all of us, even CISOs and red teamers, are incompetent.

If hospital workers spread disease because they could not be bothered to do the obvious things we -know- prevent this like basic sanitation... then yeah, I would call it negligence.

Do not put long lived cryptographic key material in the memory of an internet connected system. Ever. It is a really easy to understand rule.