Hacker News Reader

Comment by paxys | Hacker News Reader

Conveniently missing from the announcement:

- When exactly was the key leaked? Yesterday? A month ago?

- How long was it visible for? And no, "briefly" doesn't cut it.

- Did any internet traffic reach the page while it was exposed? We know you log everything, so it is a yes or no answer.

If any of these answers were pretty, I imagine they would have explicitly included them in the post.

View on HN

> How long was it visible for? And no, "briefly" doesn't cut it.

I don't know how long exactly, but in theory you can subscribe to a stream of ALL events happening at GitHub by fetching from this endpoint: https://api.github.com/events

With these events you know what new repositories are created and what changes are pushed, so you can fetch every new change. Once something gets pushed to a public repository, it's very likely that some spider will have fetched it within a few minutes.

Especially something easily identifiable as a SSH private key, you get emails from a variety of security vendors to the address associated with the commit offering their services

I thought github themselves did checks for credentials? Isn't that party of their offered services?

I’d bet that’s why it said “briefly” — someone got the alert, realized what it meant and called their boss.

> you can subscribe to a stream of ALL events happening at GitHub by fetching from this endpoint: https://api.github.com/events

Wow I am shocked that they allow "firehose" access not only for free, but without even an API key.

Given enough disk and bandwidth, does this mean you could keep your own copy of all of github? I'd love to be able to grep the whole thing.

https://www.gharchive.org/

The leaked private key should be somewhere in this archive then no?

The events stream does not contain files or anything in your git repo. Only stuff like commit messages, issues, comments, etc.

i think it does contain code

You can view it yourself, it doesn't contain the code directly, but it does contain a link to the code blob.

Wow thanks. I looked all over that page but couldn't get a rough idea of the dataset size for the event stream (not the commits).

Are we talking 1TB/year, 10TB/year, 100TB/year?

Your comment made me curious. I downloaded the complete last month (February). 28 days, every day has a gzipped json file which is on average 86MB. Summarized it is 53GB of gzipped jsons for 28 days. Roughly 700GB gzipped per year. Didn't have the chance to ungzip them all.

Indeed, and about their statement

> We have no reason to believe that the exposed key was abused and took this action out of an abundance of caution.

This is not verifiable, right? As the authentication method has no public and required revocation source, and given that the key, if having leaked, likely will be acquired by an authoritarian government, they can selectively MITM organizations and users that are not aware of this blog post.

In fairness, it's better that they rotate the key immediately before even looking at the logs. But we can demand better answers from them from this point.

Did they rotate it immediately? The only reference to a specific time is:

> This week, we discovered...

So they found out about it sometime between Monday and Thursday, and rotated it Thursday evening.

This seems a reasonable timeframe to me given the scale of the impact.

Security issues are obviously very important, but equally I would be worried if they started acting impulsively immediately after discovering the issue, because there would not have been time to understand the problem and consider the options. If you jump on a solution without giving it a bit of thought first it's too easy to exacerbate the situation.

Its not the time from discovery to rotating the key thats OPs issue. What was the time from key being public until discovery? Thats the issue. And also, while the key was public, did anyone access it?

Agreed. I don't think that was the point the comment I replied to was making though? Maybe I misread it.

Maybe I'm missing something, but since all GitHub events can be watched through the API, I don't think it particularly matters how long it was public. Anecdotal but I once accidentally pushed an AWS key (thankfully heavily locked down and not a root account) for all of 30 seconds and it was compromised anyways.

I agree. Even if its only public for one sec it should be considered compromised. However, everything is a risk management question, so knowing the amount of time it was exposed is helpful for other orgs to determine their response to this incident. Same if there was an accesslog.

The discrepancy between "briefly" and "this week" stood out to me. It reads like multiple hours or even days would classify as "briefly" here where in this context it would otherwise usually mean in the seconds or maaaaybe single-digit minutes, but even that would be stretching it.

Also sorely missing: an apology. Instead we're given crap like "out of an abundance of caution, we replaced our RSA SSH host key".

> - When exactly was the key leaked? Yesterday? A month ago?

"This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository"

That's when they discovered it had been leaked, not necessarily when the leak happened.

Reply on news.ycombinator.com