"We have no reason to believe that the exposed key was abused, but out of an abundance of caution, we are going to expose 50 million users to a potential MITM attack unless they are extremely careful."
Not a single word in the post about whether this impact was even considered when making the decision to update the key. Just swap the key, and fuck the consequences. Same with the mass password resets after a compromise that some services have done in the past years. Each of those is any phishing operation's dream come true.
Why would that be unknown? GitHub has HTTP and SSH access logs, right?
Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
Or does it log all the requests all the time as long as everything goes well, but if the wrong server crashes at the wrong time, a couple hundred requests may get lost because in the end who cares?
You'd have to have logging set up for all of these services and it would have to work over your entire CDN... and what if a CDN node crashed before it was able to submit the log entry to the log collector? You'll never know.
If train A leaves New York going 60 miles per hour with 167 people on board and train B leaves Chicago one hour later with 361 people on board going 85 miles per hour, how many people now have the key?
The answer is 31337.
To be 100% clear: Both courses of action come with associated security risks. The problem is not choosing one course of action over the other, the problem is thinking you can just skip the cost/benefit analysis because the answer is somehow 'obvious'. It's not obvious at all.
The only way a CBA would be unnecessary is if rotating the key didn't have any security risks. But it does.
- if they have evidence that the key was exposed to one person, even with zero usage of the key, failing to rotate the key is tantamount to knowingly accepting widespread compromise at a potential attacker’s whim. At GitHub’s scale, that’s untenable.
- rotating the key is the only correct reaction to that
- they should have better communications in place to help users mitigate MITM
- there really isn’t an option, because they’re critical infrastructure; I’m glad they know that and acted accordingly
- on principle this speculation makes sense, but understanding the threat makes it moot
- you hopefully know that, and it’s good to insist on thoughtful security practices but it’s important to also understand the actual risk
Thus rotating is the only logical course of action.
If you don't know for certain, you have to factor in the likelihood that it has been, and at that point, the two risks aren't equal anymore so that logic doesn't work.
You just ignore it and hope for the best ?
Only if you are certain (and better be really sure you haven't missed any cache/cdn, temp files backus etc.) it wasn't accessed you do nothing.
You should only keep moving along without key rotation if you know for 100% certainty a leak didn't happen and no one accessed the key (not theoretically impossible if they had the server logs to back it up), but anything minus that and you have to assume it's stolen.
I guess the proper way forward is a small utility that gets the latest signature through http+tls, and replaces the line in your known_hosts file, all in the background.
Looking long term, maybe we need to get rid of all the security stuff in ssh and just pipe the rest of its functionalities inside a TLS pipe. Let the os do its certificate management, reuse security bricks that are way more studied, ...
The real solution to break out of these UX/security tradeoffs is to put domain names on a blockchain: then you can simply rotate the key in your DNS record, while the blockchain model is such that you need to compromise many parties, instead of "one out of many parties", as with CAs.
Tracking Bitcoin chain for DNS updates is lightweight enough that it can be built into OS alongside other modern components such as secure enclave, TCP/IP stack and WiFi/BT/5G radios.
Even with a certificate authority model, you don't have to trust any CAs if you don't want to. Not having the option to do so is more of a problem.
Blockchain is a perfect solution to this. I wonder why it is not considered yet.
Also it's way better than TOFU, you can just add the CA key to known_hosts and avoid TOFU for each machine.
(Nevermind that you'll probably not accidentally commit some semi-ephemeral host key that's rotated often somewhere, because it will not be some special snowflake key you care about, but something handled by your infrastructure software automatically for each machine)
How easy is it to rotate the keys of your CA?
I hope you don't need that SSH connection to fix your broken CRL endpoint!
We need to compare the relative risks within the same context, namely within a company like GitHub
So it's not relevant to bring up failures of other CAs
Jump into a lesser time machine, go back to when Github were creating their SSH key, and put it into a hardware security module. Somehow share that hardware-backed security key to loads of servers over a network, without letting hackers do the same thing. Somehow get an HSM that isn't a closed-source black box from a huge defence contractor riddled with foreign spies. Somehow avoid vendor lock-in or the HSM becoming obsolete. Somehow do this when you're a scrappy startup with barely any free time.
We also have a way to put SSH host key fingerprints in DNS records already.
I'd guess that most systems aren't using DoH/DoT or end-to-end DNSSEC yet. Some browsers do, but that doesn't help tooling frequently used on the command line.
I suppose you could just accept X.509 certificates for some large/enterprise git domains, but that pokes up the hornet's nest that is CA auditing (the browser vendors are having a lot of fun with that, I'm happy that the OpenSSH devs don't have to, yet).
And where do you maintain the list that decides which hosts get to use TOFU and which ones are allowed to provide public keys? Another question very ill-fitted for the OpenSSH dev team.
There could be a well known URL that enables people to fetch ssh keys automatically in a safe manner.
So that any large entity can own your servers with easy. (Well, they already can, but not through this vulnerability.)
Anyway, the only thing CAs do is to move that prompt into another, earlier time. It's the same prompt, the same possibility for MITM, and the same amount of shared trust to get wrong. You just add a 3rd party that you have to trust.
SSH does have a CA system. Anybody that isn't managing a large datacenter will avoid it, for good reason.
Eh, let's not pretend existing SSL certificate validation is anything to write home about.
Even without any ephemeral servers involved, barely anybody is validating cert fingerprints on first use.
And among people using ephemeral servers, 99% of applications have either baked a certificate into their image (so that any compromised host means a compromise of the critical, impossible-to-revoke-or-rotate key) - or every new server gets a new cert and users have either been trained to ignore certificate warnings, or they've disabled strict host key checking and their known hosts file.
The existing SSL cert validation options are perfect if you're a home gamer or you're running a few dozen bare metal servers with all your SSL users within yelling distance in the same office. But we all know it's a joke beyond that.
There is DNS verification, but people have been trained all their lives to accept insecure DNS information (and set their systems accordingly), and I really doubt the SSH client checks the DNSSEC data.
edit: I found this helpful and honestly had no idea I should be doing this (I’m a hobbiest not a professional) https://bitlaunch.io/blog/how-to-check-your-ssh-key-fingerpr...
The problem here is that their first command that they advise using is the one that removes the old key and most users are just going to stop right there because it solves the problem of getting the key warning.
The right solution here is to provide a command that most users are going to copy and paste that deletes the old key and adds the new key all at once.
If your system doesn't know the public key of an SSH server, when you connect the first time, the SSH client will display a warning and ask you if you accept the server key. An attacker could be between you and Github and if you accept without checking it's the correct key, you would be toast.
Meaning that a lot of users will blindly accept whatever new key (even when it might be the one owened by attacker doing MITM) because Github, their college or random person on internet said that that's what you have to do to get rid of error.
This is less likely because unlike for TOFU the SSH client just rejects the mismatch and insists you take manual action, and the likely manual action will be "Paste this stuff from the announcement".
So an adversary needs to either subvert whatever messaging you see (which is tricky, likely impossible for a random user visiting the github web site wondering what's wrong) or hope that you just try to muddle along and do TOFU again, putting you in the same spot as a whole bunch of users every day at GitHub.
If the key was breached and Github just didn't know it, then a breach happened, then only Github would be to blame.
If Github rotates its key, and somebody suffers a MITM attack, the blame is more diffuse. Why didn't they verify the key out of band?
The MITM-at-the-start risk is of course real, but I think this new everyone -restarts-simultaneously risk is qualitatively different enough to be worth at least considering.
One way to solve this in TOFU is to have a time window where both keys are presented.
well, yes
github doesn't accept https push any more
Maybe you’re referring to how they no longer accept passwords for HTTPS auth? You have to auth for HTTPS push with a personal access token.
I get a bit paranoid when having to deal with Tokens on various CI/CD environments as it stands. And the things that start breaking every year when I forget to update them. Note: this is personal/hobby projects, not corporate stuff, where I'm strictly in the codebase and try to keep my fingers out of CI/CD beyond getting a Docker image built, and someone else configures the keys/auth.
I sure did. Doesn’t everyone?
So MITM for some of 50m users is strictly better than MITM for all of 50m users.
How do you know this?
Github runs scanner for private keys in public and private repos and notifies owner (I did it once so I know ... ;)). So some Github engineer likely would have received such an email if what you say is true. Hilarious.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
Hilarious.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
> leaked in public repo
Me: Yeah, that's why they are doing it.
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s (RSA)
SHA256:br9IjFspm1vxR3iA35FWE+4VTyz1hYVLIE2t1/CeyWQ (DSA - deprecated)
SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM (ECDSA)
SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU (Ed25519)We humans really aren't cut out for this, are we.
Generally speaking, one would not consider an internet comment directing folks to GitHub's actual SSH fingerprints a "man in the middle" as the phrase in this context usually has a negative implication, where in this case defanor is in fact simply mirroring the actual information that GitHub has officially posted in a way that is much more helpful than yetanotherjosh's "double check it is the expected value". For most of us idiots, we don't know what the expected value is!
So thank you defanor for sharing, and thank you darthrupert for asking for clarification. Y'all contributed to educating myself and others and now we know more because of it.
Good clarifications everywhere, yes.
(Not genuinely concerned about this risk, but "Reflections on Trusting Trust" reverberates.)
Not all of us are familiar enough with the SSH protocol to understand how to "double check the expected value"? Where can I determine what the expected value should be?
It should error like this:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
It would just take many ages of the universe, at present, to calculate a collision, right?
If you somehow can MITM an SSH connection on the first connection, you can probably use any key. Most people don't check the fingerprint.
But you are correct, computing an SSH key with a collisionwis expected to take an infeasible amount of time/energy with current understanding of crypto and available computers.
Or the parts below it about updating and verifying in other ways.
https://docs.github.com/en/authentication/keeping-your-accou...
Yes, this assumes the github-hosted docs and your SSL connection to them are not also compromised, but it's far better than not checking at all.
What do I have to do to ensure I connected to the right server?? I thought just making sure the correct RSA key was in known_hosts would be enough?
If you read the blog post on a verified domain and saw the new key and updated manually, or you deleted the known key and verified the key fingerprint when it warned you about an unknown key, you should be good to go. Here, you trust the people who issue TLS certificates and you trust github to be in control of their domain name, so you can be reasonably confident that the key they advertised on their website is the correct key. If your internet connection was compromised, you would have got an error message when you connected to https://github.blog (because they wouldn't have a certificate from a trusted certificate issuer) or when you connected to the git server (because they wouldn't have the key you just trusted).
If you saw the blog post and then removed the old key and told ssh to save the new key it's receiving without checking it matches the value on the webpage, you might have a problem. The connection to github's ssl could have been compromised, and if you just accepted whatever it told you, you have no trusted intermediate to verify that the key is trustworthy. All you know is that each time you connect to github's server hereafter, you're either connecting to a server with the same key (no error), or you're connecting to one that doesn't have the same key (error message). But whether you can trust that key? You don't know that. You just know it's the same key.
But even if you did the latter, all is not lost. You can look at the known_hosts file (on Linux and MacOS it's ~/.ssh/known_hosts) and check the fingerprint. If it's what they advertise, then you're good to go. If it's different, you should fix it and find people who can help you deal with a security incident.
The reason people are raising a flag is that today, lots of people will be rotating their key. That means if you're looking to target someone, today is the day to do it. Even if 90% of people do it the proper way, by manually verifying the key, that still means there's going to be a lot of people who could be victimised today.
I assume it's safe because the SSL cert for docs.github.com is probably not compromised, so it's giving us the right key, and compromising docs.github.com would be extra effort and is unlikely to happen.
However, I wonder what kind of steps an MITM attack would have to perform, I assume one of the easiest would be compromising my local DNS server, since regular DNS does not offer a high level of security, then github.com resolves to the attacker's IP and the attack works. Do you have examples of such attacks that don't involve a virus already active on the end user's PC? Maybe if someone owns an IP previously owned by Github that is still somehow advertised as being Github by some DNS lagging behing?
The best practice is to verify the fingerprint out of band using a secure channel. In this case, that's HTTPS and docs.github.com. If (hypothetically) docs.github.com was also compromised, then you don't have a secure channel.
https://en.m.wikipedia.org/wiki/Man-in-the-middle_attack has some MITM examples.
We should have a blockchain for certificates btw. That would be such an amazing solution to this problem. You could advertise ahead of the time that you are changing certificates and we could verify that it was in fact you.