Reproducible Git Bundles | Hacker News Reader

Reproducible Git Bundles

baecher.dev

187 pointspbhn34 comments

Parallelism doesn't inherently have to break reproducibility. If the output depends on the scheduling order of the threads, then it will. But it's possible to use threads to farm out work while still doing the same work deterministically.

Strictly speaking in terms of system backups, this approach misses the git reflog which is a crucial "get out of jail free" card on the (hopefully) rare occasion that you actually need it.

The question is whether you can expect this format to stay stable and reproducible across git versions. Remember the fallout from git 2.38 when the output of 'git archive' changed. Although for this backup use case it would just mean the next backup with a new format would make a full copy once.

git bundles have a standardised format (defined in [1]) while git archives did not and continue to not have one. Git can still handle all previous bundle versions and you can specify which version of the bundle standard you want to use when creating bundles.

So no the git 2.38 issue should not be a problem for bundles even if the format changes in the future.

1. https://git-scm.com/docs/gitformat-bundle

Standardized format != guaranteed reproducible. Git makes no promises that it'll keep PACK contents stable, just that they're guaranteed to "deflate" to the same contents.

Which is what the linked article discovered. Threading is a trivial way to discover this, but there's other ways PACK contents might differ across versions.

Yeah, even tar is not stable bit for bit across versions.

Absolute stability is not needed for this use case. Just reproducible per git version would be enough, to ensure mostly no redundant backups.

Why not just use tar or any other archive tool on the repository .git folder? Unless your repository is a un-gc'd mess with millions of unpacked objects...

I think that is a fair alternative, but restoring the backup means that the repository is in a bit of a weird state. Whereas a bundle can be cloned from nicely. Your way does have the property that it includes hooks and config, though (which could be desired or not).

There are very few/no operations that will put a git repository in a "weird" state.

Actually, any snapshot of a Git repository is consistent (due to its CAS nature), minus the index which doesn't really need backup anyway.

This is also non-deterministic between versions of tar, but I guess for this usecase that would be fine. It’s just not good for reproducible build systems when trying to recreate tarballs after years.

Does not meet stated goal of the author:

> The naive solution of simply backing up the entire file-system tree is clearly not desirable since that would clutter the backup with useless build artifacts.

Build artifacts can be filtered out with tar --exclude patterns, but this is a language-dependent set that will require curation.

Wouldn't all these useless build artifacts be outside the .git folders?

git bundle is the answer to the question "I need to keep my development trees in sync on different computers, with no network connection between them." https://stackoverflow.com/q/3635952/308851 buried in comments is a great script automating the process https://github.com/cxw42/git-tools/blob/master/wormhole alas it's abandoned but still, it's pretty great.

An engaging post and a lovely piece of sleuthing. Thanks!

Feels like bundles is a bad idea for back ups. Perhaps --bare repo in zip archive would be better

>The naive solution of simply backing up the entire file-system tree is clearly not desirable since that would clutter the backup with useless build artifacts.

Just ignore the files in .gitignore and backup the entire file-system tree.

Don’t be clever. This is a backup of source code that took many hours/days/weeks of effort to create. Since git is mainly source code, it is not that big of space hog.

Disk space is cheap. Time is not.

> Disk space is cheap. Time is not.

At the same time, this train of though is why VSCode dumps 8 GiB into the ~/.config directory, causing not only a lot longer time to wait for any backup to finish, but also incurring time in getting more and more backup disks, figuring out where to physically store them, etc. Better spend more time up front getting the storage right, and save space and time for those that use it later.

> The naive solution of simply backing up the entire file-system tree is clearly not desirable since that would clutter the backup with useless build artifacts.

`git rm -r --cached .`

> One solution is to create a fresh clone (with --mirror), but that will typically consist of many small files which isn't ideal for backups, either.

this is precisely what tar(1) was made for..

Do you mean `git clean`?

Reply on news.ycombinator.com