Github Archive | Hacker News Reader

Github Archive

githubarchive.org

38 pointsig17 comments

Dumb question, but is the code behind the archive open source? I didn't see any link but did see the star button which...

Ok...I see the star button serves as a link https://github.com/igrigorik/githubarchive.org/

It's great to have other crawler projects to learn from

Yep, the repository is public and a big chunk of this public data is accessible on BigQuery for anyone to query and analyze.

Ilya Grigorik and myself just gave a talk at Strata[1] on interesting analysis and visualizations that people did with the GitHub Archive as part of the GitHub data challenge[2].

We'll post a link to the talk soon. We announced during the talk that BigQuery will soon (in the next day or two) have an archive of the GitHub public timeline from February 2011 to now. I can't wait to see what people do with it!

[1] http://strataconf.com/stratany2012/public/schedule/speaker/2...

[2] https://github.com/blog/1162-github-data-challenge-winners

It is a great project. However, it's not based on crawling, as far as I understand: Github pushes their data to Google's BigQuery tool, which this gets its data from.

Correction: We (GitHub) do not push data to BigQuery. Ilya's project archives the GitHub public timeline. That data is then uploaded from his archive into BigQuery. The BigQuery project is public, so anyone can use it.

Is there any docs on the github public timeline? I did found http://github.com/timeline which made me guess for http://github.com/timeline.json which works, but is this documented anywhere? Are there parameters? How often does it refresh? Thnx!

I haven't had to deal with this scenario, thankfully, but I seem to recall dimly that there is some mechanism to rescue GitHub users who accidentally publish passwords and access keys. Does the archive respect this same mechanism? (Or did I dream up the mechanism in the first place?)

There certainly is a mechanism [0], however it doesn't actually completely remove the commit from the remote GitHub repo, as people with the direct link can still access it. I'll check to see if I can find removed commits using this.

[0] https://help.github.com/articles/remove-sensitive-data

Reply on news.ycombinator.com