Hacker News Reader

Comment by low_tech_punk | Hacker News Reader

On a philosophical note, git, being a distributed source control system has no limits to repo sizing, but github, being is a centralized "hub" suffers scaling problems. It's fascinating to watch these tradeoffs play out. Maybe, IPFS would come to the rescue one day.

View on HN

I don't understand where git's distributed nature comes into play here. Perforce is fully centralized, and it also doesn't have any limits to repo size - as long as you have enough disk, P4 will handle it.

In fact, Git's distributed nature actually makes it infamously bad at scaling with repo size - since it requires every "participant" in a repo to have a copy of the entire repo before they can start any work on it.

This is where something like VFSForGit [0] helps out. Instead of cloning the entire repo, it creates a virtual file system and fetches objects on demand. MSFT uses it internally for the Windows source tree (which now exceeds 300GB).

[0]: https://github.com/microsoft/VFSForGit

VFSForGit seems to be in maintenance mode now - do you know what it was replaced by, and if there's something that works cross-platform?

It was replaced by Scalar and is now merged into Git:

Introduced: https://devblogs.microsoft.com/devops/introducing-scalar/

Integration into Git: https://github.blog/2022-10-13-the-story-of-scalar/

Shallow cloning of Git repos is a thing. Basically, you get a fake head commit (that includes all the files) and none of the real history. Useful if you only intend to build, or make changes locally. If you want to push, you have to unshallow first.

would be cool to be able to commit without unshallowing first

Not possible, due to the way how git works: It's a merkle tree of commits, where each of these commits point to a file tree (content-addressed by the hash) and the previous commit

No, creating a git commit merely requires knowing the current state of all the files and the hash of the previous commit. You don't need the actual contents of the previous commits.

https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Pushing from a shallow clone to a remote is more complex, but is supported in modern git versions.

https://stackoverflow.com/a/6900428

I believe the problem here is related to LLVM using a CI model, where all changes eventually go to a single tree and tests are run on that single tree. Hosting that CI happen on another site than github wouldn't really change anything. Maybe change the limit, but all such CI hosts have a limit, for the same reason that all the search engines have a size limit for web pages they scrape.

The problem here is unbounded growth of a git repo. In this specific case, a size limit was triggered. In other circumstances, it would have required too much time to transfer or no storage space left.

Anyway, the problem is that git stores all changes, forever. A better approach would be to clean old commits, or somehow merge them into snapshots of fixed timespans (like, anything older than a year get compressed into monthly changesets)

Git allows you to rewrite history, so you can "crush" old commits to reduce the size of history as needed.

Sure, but rewriting it manually is a tedious process. Should be automated on github side, to keep repo size approx constant over time.

That would be silent data loss, so absolutely should not be automated.

Github cant rewrite the refs on their own without breaking users stuff. They can only repack the existing objects, the squashing needs to be done by the developers. Also its a non-fast-forward thing, so it needs to be coordinated between the git users anyways.

I think this is what `git filter-branch` its supposed to be for: https://git-scm.com/docs/git-filter-branch

I've never used it before, but from what I understand, it's very powerful but also very confusing and easy to mess up, and of course with a sort of vague ambiguous name that makes it hard to discover; in other words, it's quintessentially git.

It would be great to have some kind of way to do this while still maintaining the merkle-tree.

Isn’t that what packs are for? The raw, content addressable object store has no inherent optimization for reducing repo size. Any changed file is completely copied until a higher level does something to compress that down.

This is not a "problem", this is why source control exists. You should never rewrite published history.

A value of a commit approaches zero as it gets older. After a certain threshold, no one will ever see it. Never say never; any reason why should we keep deadweight around?

Your premise is incorrect. The other day I was looking around a repository that's been through many migrations, and found a commit from 2004 that was relevant to my interests.

As long as a line code is in use, there is value in knowing who and when it was authored.

If a 10 year old vulnerability is found in OpenSSL, it be nice to be able to investigate if it was an accident or an act of espionage.

A more conservative approach would be some sort of layered storage/archiving, I guess. The older the commit, the less likely it is to be used, so it could be archive in a different storage, optimized for long term. This way you keep the "hot" history small, while keeping the full history still available.

That's generally how git packs are used at large organizations that host their own repositories. I'm sure GitHub does something similar.

I don't think that your points are actually in conflict.

If this is my source code, I want the whole history. I want that 10-year old commit that isn't used in any current branch. A build machine may not need any history: it just wants to check out a particular branch as it is right now, and that works too.

But there is an intermediate case: Let's say that I have an issue with a dependency. I might check out that code and want some history to know what has changed recently, but I don't need that huge zip file that was accidentally checked in and then removed 4 years ago. If it were a consistent problem, perhaps you'd invent some sort of 'shallow' or 'partial' clone, something like this:

https://github.blog/2020-12-21-get-up-to-speed-with-partial-...

True, though shallow clones have performance issues: https://github.com/Homebrew/discussions/discussions/225

I don't think I can agree with that.

Accidentally publish secrets/credentials? Rotate them yes but also remove them from the published history.

Accidentally publish a binary for a build tool without the proper license? Definitely remove it (and add it to your .gitignore so it doesn't happen again!)

You discover a major flaw that bricks certain systems or causes data loss? Retroactively replace the Makefile/configure script/whatever to print out a warning instead of building the bad build.

I'm sure there are others.

AFAIK, copyright problems are fixed by just another commit without history rewrite. There's also no need to care about outdated credentials. Should bugs be fixed by deleting all history too? That code was buggy, bad, bad, must delete? VCS just becomes glorified ftp this way.

Reply on news.ycombinator.com