Hacker News Reader

Comment by orf | Hacker News Reader

Do you have some GitHub architectural knowledge you’d like to share with us?

A push pushes commits and blobs and trees and tags. It’s an interesting metric to track, but the core unit of complexity (and expense) worth tracking on GitHub’s side is obviously the commit.

There’s a difference between pushing 1 commit and 100.

View on HN

> There’s a difference between pushing 1 commit and 100.

There isn’t much. GitHub doesn’t run actions separately for each commit. It runs them on pushes. I’m trying to think of a thing that would happen for each commit in each push and coming up blank.

It does things like scan for references to issues to index, but it would just scan the log for a range.

I did disagree with GP though because there is no reason to assume that the ratio of commits to pushes has materially changed. So if that is the proxy they have always used for measuring growth, and they know it reliably does that then I think it’s a reasonable way to communicate this to this audience.

> It runs them on pushes

Sure, because pushes are how you update a reference. That’s really what triggers an action: a reference changing. And there could be a bunch of those in a push.

A commit costs storage, you’ve got secret scanning, it needs to be indexed in a way that can be referenced in commit messages and comments, a commit message itself can close issues or reference other PRs, stored and served individually and immediately via the web UI or git clients, etc etc.

It’s also like… the core unit of git.

None of the things you mention - indexing or secret scan would be done individually for each commit. As I already said, this would be a log of all commits in the range pushed - it would be scanned once for those things. There is no need for a loop running over a range of commits and processing each one.

There 100% is at least for things like secret scanning and message parsing.

Secret scanning needs to make sure my repo as a whole has no secrets. It’s not acceptable to have 1 commit introducing it and 1 removing it because the secret is still recoverable.

Every commit is also surely an entry in a database somewhere. I can navigate in GitHub directly to any individual commit so there is definitely some overhead of some type.

You are right about secret scanning, but its worth noting that is only enabled by default on public repositories. It is an extra paid feature for organizations on a teams or enterprise subscription, and isn't available at all for individual owned repositories outside of an enterprise subscription.

And yes, I agree there is indexing of commits, but that is a batch insert from a log.

It's not necessary to perform secret scanning on a per-commit basis. The most efficient way is to just scan all blob objects being pushed; there is no reason to even be aware of an object's location (tree path or commit) except for diagnostic messages.

> I can navigate in GitHub directly to any individual commit

You can do the same with the git command line client. The overhead you claim is already in the git on-disk format. Github might very well duplicate this information in a database somewhere, but it doesn't follow from your observation.

Reply on news.ycombinator.com