Hacker News Reader

Comment by tremon | Hacker News Reader

Why is Github talking about number of commits here, and not pushes? Are there a lot of tools/people using github as an online editing platform?

View on HN

GH processes at the commit level for things (including actions) even though they're bundled in a push... it's relevant to the load on their infrastructure.

But they can't process the commit until I push it?

A push containing 100 commits is more expensive than a push containing 1 commit.

But is it 100x more expensive?

Probably? If you do 100 pushes instead, there is roughly zero additional data. At best you'd be comparing cache costs, which probably are lper for one large push, but there's a ton of calculation and CI that runs per commit regardless of other data being cached.

Why don’t you have your actions run on pr or push instead of on commit? Why would you even want that? I’ve never seen actions set up that way. If I push a branch with 100 commits then it’ll only run CI once. It’ll show the rest of the commits in the UI, sure, but that doesn’t mean that it’s the same performance impact. It could very well be 1 db transaction with multiple rows written instead of 100. I think you’re reducing this problem too much without knowing their architecture.

Github runs CI per PR push (obviously), and per main-branch commit (click on commit history in any project with CI, see a build result check mark on each one - that's true if you push a dozen commits too (I've done that)), in nearly all setups I've seen. I'm not sure how much of that is required vs default though.

With enough effort, you can rather obviously run CI per PR commit (it's a programmable system), but I've never seen aUI-integrated way to track the results, aside from browsing custom job names, which is very far from what I'd call "integrated" when compared to PR-level build markers. Similarly, I'm not aware of (but would not be surprised by) any way to disable per-main-branch commit builds, aside from initial pushes.

But I haven't poked around deeply in the settings, and business-account settings are rather different anyway so those might be wildly different / more flexible / more obtuse in exciting ways. Github is a very large and complicated product at this point, darn near anything could exist if you dive through enough UI layers or use old URLs to find soft-deprecated features.

Also, honestly, 100 commits = 1 transaction? That's far more of an over-simplification than anything I've said. It's a massive product with thousands of engineers, there's no chance at all it's just one database.

Things like password scanning have to look per object uploaded

In what way? If I have multiple local commits pushed once I expect to see CI type actions to run once for the push rather than for ever commit in the push.

This is configurable in Github Actions. Many projects want to run CI for each commit to avoid situations where one commit breaks the build and the commit after in the same push or PR fixes it. Broken commits in the history makes bisecting harder.

Interesting, never realized that was an option. Thanks!

Bigger numbers sound more impressive.

"Our billion-dollar infrastructure crumbles under a tremendous flood of 50 PRs per second" would just sound embarrassing.

I'm not following your line of questioning. Without ever using github as an online editing platform, you can do one push with two new commits.

Commits are not expensive, pushes are. You can do any number of commits before you do one push, unless you are editing online, in which case every act is it's own commit & push.

You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.

>You can rig up a local ide to pathologically commit+push per save

The dev system we use for a 3rd party hosting provider (a big one) requires a commit and push for every file save while we're developing. I created a build system for this that copies the whole repo to a temp folder. As we save changes to files in the main repo folder, the build system watches for changes and copies the changed file to the temp folder, then does a commit on the temp folder and pushes to a an intermediary repo in github which then triggers an action that causes the 3rd party system to update from the intermediary repo. This way we don't pollute our main source repo with a commit every time we save an update to a source file.

It's not my favorite way to develop but it's caused us no real problems except when github goes down.

> requires a commit and push for every file save

I don’t think I could imagine a stupider idea than this if I tried. To paraphrase Babbage: I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a solution.

meh, it's just external undo button. It's useful. It might or might not be worth the cost, but it's not that it delivers no value or causes some harm (other than cost/reliability)

https://delta.dev/

Do you have some GitHub architectural knowledge you’d like to share with us?

A push pushes commits and blobs and trees and tags. It’s an interesting metric to track, but the core unit of complexity (and expense) worth tracking on GitHub’s side is obviously the commit.

There’s a difference between pushing 1 commit and 100.

> There’s a difference between pushing 1 commit and 100.

There isn’t much. GitHub doesn’t run actions separately for each commit. It runs them on pushes. I’m trying to think of a thing that would happen for each commit in each push and coming up blank.

It does things like scan for references to issues to index, but it would just scan the log for a range.

I did disagree with GP though because there is no reason to assume that the ratio of commits to pushes has materially changed. So if that is the proxy they have always used for measuring growth, and they know it reliably does that then I think it’s a reasonable way to communicate this to this audience.

> It runs them on pushes

Sure, because pushes are how you update a reference. That’s really what triggers an action: a reference changing. And there could be a bunch of those in a push.

A commit costs storage, you’ve got secret scanning, it needs to be indexed in a way that can be referenced in commit messages and comments, a commit message itself can close issues or reference other PRs, stored and served individually and immediately via the web UI or git clients, etc etc.

It’s also like… the core unit of git.

None of the things you mention - indexing or secret scan would be done individually for each commit. As I already said, this would be a log of all commits in the range pushed - it would be scanned once for those things. There is no need for a loop running over a range of commits and processing each one.

There 100% is at least for things like secret scanning and message parsing.

Secret scanning needs to make sure my repo as a whole has no secrets. It’s not acceptable to have 1 commit introducing it and 1 removing it because the secret is still recoverable.

Every commit is also surely an entry in a database somewhere. I can navigate in GitHub directly to any individual commit so there is definitely some overhead of some type.

You are right about secret scanning, but its worth noting that is only enabled by default on public repositories. It is an extra paid feature for organizations on a teams or enterprise subscription, and isn't available at all for individual owned repositories outside of an enterprise subscription.

And yes, I agree there is indexing of commits, but that is a batch insert from a log.

It's not necessary to perform secret scanning on a per-commit basis. The most efficient way is to just scan all blob objects being pushed; there is no reason to even be aware of an object's location (tree path or commit) except for diagnostic messages.

> I can navigate in GitHub directly to any individual commit

You can do the same with the git command line client. The overhead you claim is already in the git on-disk format. Github might very well duplicate this information in a database somewhere, but it doesn't follow from your observation.

Exactly. You can have 3000 commits in a branch, and unless you don't push each of them one by one, shouldn't be a problem the number.

Reply on news.ycombinator.com