Hacker News Reader

> you can subscribe to a stream of ALL events happening at GitHub by fetching from this endpoint: https://api.github.com/events

Wow I am shocked that they allow "firehose" access not only for free, but without even an API key.

Given enough disk and bandwidth, does this mean you could keep your own copy of all of github? I'd love to be able to grep the whole thing.

View on HN

https://www.gharchive.org/

The leaked private key should be somewhere in this archive then no?

The events stream does not contain files or anything in your git repo. Only stuff like commit messages, issues, comments, etc.

i think it does contain code

You can view it yourself, it doesn't contain the code directly, but it does contain a link to the code blob.

Wow thanks. I looked all over that page but couldn't get a rough idea of the dataset size for the event stream (not the commits).

Are we talking 1TB/year, 10TB/year, 100TB/year?

Your comment made me curious. I downloaded the complete last month (February). 28 days, every day has a gzipped json file which is on average 86MB. Summarized it is 53GB of gzipped jsons for 28 days. Roughly 700GB gzipped per year. Didn't have the chance to ungzip them all.

Reply on news.ycombinator.com