Git filter-repo: much easier/faster alternative to filter-branch | Hacker News Reader

Git filter-repo: much easier/faster alternative to filter-branch

github.com

60 pointsdmart9 comments

I did something similar a couple years ago. I needed to rewrite an entire repo's history so that I could reformat all the code _and_ modernize all the JS syntax throughout every commit in the repo's history (dating back to 2013).

I started with `filter-branch`, but found it way too slow (especially on a corporate-controlled Windows machine, where starting up additional processes seems to have a lot of overhead). I concluded that I needed to run the entire filtering logic in a single process to avoid that overhead. Started writing my own with `pygit2`, but then found a repo called `pylter-branch` which did most of that same "loop through commits and reprocess" work for me - I just had to add a lot of additional logic on top for the specific reprocessing I wanted to do.

Ended up being able to reprocess about 15K commits in around 4.5 hours. Given the amount of processing I was doing, that was pretty good.

I did an extensive writeup [1] on the problem statement, investigation, and techniques I used if anyone's interested.

[0] https://github.com/sergelevin/pylter-branch

[1] https://blog.isquaredsoftware.com/2018/11/git-js-history-rew...

Why did your group want to do a refacctor like that retrospectively rather than just checking in those changes on a forward going basis?

Heh. It wasn't even the rest of my team - I'm the only one who cared enough to do anything about it :)

Per the post, I wanted to auto-reformat the entire codebase, as well as codemod the JS source to update it to more modern syntax (ES modules instead of AMD, `const/let` instead of `var`, shorthand object literals, etc). But, that would have left most of the codebase history showing that the last commit touching a line was "MARK REFORMATTED ALL THE THINGS".

Since I was rewriting the entire repo history to strip out all the junk files that were consuming space, which was going to create an entirely new alternate history anyway, I figured I might as well take advantage of the opportunity to redo all the source as well:

https://blog.isquaredsoftware.com/2018/11/git-js-history-rew...

> But, that would have left most of the codebase history showing that the last commit touching a line was "MARK REFORMATTED ALL THE THINGS".

Indeed, this is a good reason to do as you chose. I hadn't thought of it (even though I have done the same to a code base).

> If commit messages refer to other commits by ID (e.g. "this reverts commit 01234567890abcdef", "In commit 0013deadbeef9a..."), those commit messages should be rewritten to refer to the new commit IDs

Finally, a tool that does this! I wish git rebase could do it.

Git filter-repo is mainly from a single author newren, who also is a significant contributor to git itself: https://github.com/git/git/graphs/contributors The repo seems open to PR's though, given the many commits from one-time contributors.

[Edited] A question: how would one check the integrity of the resulting git repo (apart from an obvious rsync check of the source files in master)? Say, I would like to create a branch from an older commit and rewrite its history with "git rebase -i"; that could fail on a a repo without full integrity.

I used this earlier this year to split a monorepo into a submodule + 4 other repos. It made it incredibly easy to rewrite history to move files going to different repos, then remove the history of any files not staying in a particular repo.

And it was fast. Also, being able to pass in files containing rewrite rules allowed me to easily do multiple dry-runs of the code split ahead of time so I was able to minimize the code freeze for our team.

This looks so much nicer than filter-branch, which is slow beyond belief.

We've used bfg (which at least performs in reasonable time) to filter out files containing viruses or security tokens/passwords, but there are some definite limitations.

Wow these are really interesting projects

Reply on news.ycombinator.com