Cleaning Up Git History
blog.sulami.xyz
blog.sulami.xyz
> Crafting commits is hard if one wants perfect commits and most of us don't have time to do that.
This is a common sentiment, but I really think that this is a product of the tooling instead of a fundamental difficulty of the task. If it wasn't such a pita to split, join, and shift commits around then people would consider trying to do it more often.
Another comment:
> TIL: Splitting A Commit
> git rebase -i <ref> Mark commit as 'edit' Continue with rebase, it will stop on 'edit' marked commit git reset HEAD git add --patch git commit git rebase --continue
What is the mountain of context that someone needs to know before even that "one simple trick"-style explanation would make any sense at all? How would you describe or recommend that process to someone in a technical non-dev role that has never used source control? You wouldn't. So they'll never split a commit. So now you have an avalanche of users that don't even approach the problem of crafting a PR with the mindset to produce clean commits, so it can never become the norm.
At the least we need a more direct way to accomplish three main things: split a commit, merge adjacent commits into one, reorder commits. If these "basic operations" on commit chains are easy to accomplish we might have a chance at having a clean git clean history.
That’s the audience though. I know technical non-devs in various roles that would kill for the kind of features that git provides. They’re stuck manually versioning files. There is definitely a market out there for something like this but even simple version control is very tricky for non-devs. It’s just something we’re more intitivively good at probably to do with how we approach breaking down problems. Animators, graphic designers, architects etc tend to think more holistically. If they use these tools at all they’re usually a service provided by some in house experts, such as the engineering team or IT.
Everywhere I've worked has used a similar flow. Commits in master correspond 1:1 to PRs. Your personal flow of local commits or branches is irrelevant; your PR is reviewed as a unit, and merged as a squashed commit on top of master. The description of the PR is either exactly identical to the commit message, or only slightly modified.
Clearly lots of people don't work that way, because that doesn't leave any room for a messy git history. I also don't know why I should care about the in-progress commits of my coworkers when I'm reviewing a PR as a unit.
I mean commits in need of repair happens, however good your workflow is.
I've really just never needed to do major repairs to the master branch of a project I was working on. We're probably missing out on git features that could be nifty, but it's also easier to avoid shooting ourselves in the foot.
Not to mention how do you revert commits on master? A big PR that broke prod will lead to nastyy merge conflicts when reverting.
I know there's a case for larger feature branches that accumulate an entire large feature before being merged into master, but in a world of microservices, it's just not a use case we've had to consider.
And reverting PRs happens occasionally, but because changes are much smaller than the scale of what I assume you deal with, we never have many (if any) merge conflicts to deal with. We primarily deploy new features with feature flags, so it's vastly easier to just turn them off until we can fix the issue.
I think you are tempering your stance in the name of arguing charitably, but I’ll push against that.
No, there is no case for this. Branches are pain accumulators. The longer they live, the more pain they will bring to everyone involved.
If you need to work on something big, that’s what feature flags are for.
I don't see the value for small changes, just squash. I also don't see the value in a PR-based workflow, as your team's PR practices should have sufficient context without asking everyone to rewrite history.
I also don't really believe that devs can reliably retcon atomic commits that compile using just git commands. You make a nice commit for the new file but forget that it has a dependency in another new file. Or you remember the dependency and painstakingly add -p the functions necessary for the commit, but forget one variable definition that was in a chunk at the top of the file.
"Good commits" are one of those things that benefit the whole, but no individual developer is incentivized to do. Because of that, it's easier to enforce at code review (rather than hope developers just do it). Many companies do this by squashing changes, so a developers' commits and the commits on trunk are decoupled.
I think the best way to make changes (and therefore commits on trunk) understandable is to split them up into series of dependent changes; something supported by the code review tooling used at places like FB / Uber / Google, but not super well in GitHub... Which leads to large PRs with a lot of unrelated fixes.
Disclaimer: I work at a company trying to bring that dependent change / stacked diff workflow to git and GitHub (https://graphite.dev).
GitHub I understand, the tooling here is terrible. I have my own scripts to solve this, but there are fundamental issues with how GitHub treats branches. But git? Huh?
The "stacked diff" workflow is more commonly known as a patch set or patch series, and is as old as git itself. See the LKML for examples. Tools like Gerrit or Sourcehut promote this workflow as a first-class citizen. It's how you use git over a mailing list anyways.
However, it's in contradiction to TDD. If you have a bug (in master) the first thing you are supposed to do is to create a test that fails because of the bug. This is a self-contained commit, but it will not build (assuming your test are run during the build as they should). The next commit is fixing the bug and now it builds again.
So the commit with the new test should be marked as unsuitable for bisecting. How to do that easily in away that bisecting remains easy I have yet to work out.
If you want to work like that though one way to do it is to work in a branch/PR - building out your commit that fails, then your iterative commits - and then squash the branch into a single commit onto main layer on.
If anyone want to see your TDD tiny commits process later they can go and see it in the PR.
In that case, you can have a merge request with a commit to add a test and then another commit to fix it. As long as both commits build, bisect will still work.
I agree with having CI enforce both passing tests and working builds, but why would you want the build step to require passing tests?
(ETA: It's easy to forget that the I in CI stands for "integration" and was always focused on merges of changes more than specific changes.)
Then I realized that if I bundle the documentation change itself into the same commit I can have the same benefit while also creating documentation that people can read without digging through the commit log to find it!
These days my commit messages are much shorter, but the commits usually bundle implementation, tests and documentation updates into the same unit.
I also make sure my commits link out to a relevant issue. This gives me somewhere I can record additional context and drop in things like screenshots that help illustrate my thought process while working on the code.
git rebase -i <ref> Mark commit as 'edit' Continue with rebase, it will stop on 'edit' marked commit git reset HEAD git add --patch git commit git rebase --continue
I saw a lot of people merging branches a single commit "to keep the history clean" while in fact it just made the history less legible and less reversible.
This way, the history would be meaningful instead of it be littered by fix up commits and it makes reverts easy.
Why would someone do that? I’d argue that if there is cleaning to be done, it should be done way before something is ready to be merged, while context is fresh in your head.
Also, cleaning before something is ready to be merged will lead to sloppy history removal as, often, people are eager to merge for a variety of reasons, and rewriting history may not be seen as a priority.
I guess it's personal preference, I don't recall losing context, as I don't let too much of time passed. And I actually ensure nice commits are written before code review otherwise it might be harder for the reviewer to get the context of the changes.
Of course you have to consider what history is relevant and create relevant commits for those. But a WIP commit at the end of the day is rarely more relevant when split up from the finishing up of that work the next morning.
git bisect --first-parent
The first interest is what you said, for every commit to build.
The second interest is to help prevent loss during development. If I am working on something and need to finish for the day, but haven't yet got the code to a working state, I want to be able to commit and push my code in case something happens to my laptop.
Also, sometimes I am making a series of big changes, but I need to make them all for the code to work. I want to have checkpoint along the way, in case I need to undo something (or even just see what I have changed since the checkpoint!) committing WIP code that doesn't build lets you do that.
We usually solve this problem with squash merges, so every commit on the main branch is the full working feature, but there are downsides to that technique.
At first glance, it seems like a beneficial goal. If I ever need to checkout different points of history to find a breaking change, it'd be nice if I can run the code after every `git checkout`.
The process I've set up for my team allows us to get the best of both worlds, in my opinion, with almost no extra effort:
1. Create a branch and open a pull request. Nothing gets pushed directly to master.
2. After the pull request has been tested and reviewed, we Squash and Merge into master on GitHub with a single button click.
This way we end up with a single commit in master for each feature, bugfix, etc. There's also a link to the PR in the commit message where you can view the individual commits that were squashed as well as links to any related Jira issues.
Squash/rebase also plays horrendously with my local branches. If I make a PR, I should be able to use `git branch -d branch_name` to safely verify that the branch has been merged into main and can be deleted. If you squash/rebase instead, then I need to use the unconditional `git branch -D branch_name`. git can no longer verify that I'm performing a safe operation, because the history needed to determine whether it is safe has been rewritten.
This is a very good point, and it has been slightly annoying at times with this approach. It hasn't been a major pain point for me, so I've just dealt with it. I've seen some scripts/aliases that claim to solve this, but I haven't spent much time looking into it.
Take a look at patches in the LKML and their associated revisions. This is the most sane approach to history. Changes are broken up logically, nearly all commits in a patch series are individually functional/correct, and in their presented order they tell a story for the given feature.
Your "update" or "whoops" or "wip" or "address feedback" commits are, almost universally, entirely useless. There's no point in checking out one of those commits. They don't help tell a narrative to uniformed readers. They're just noise.
If I implement text-box with a save button there is no way I can functionally commit "working change" where I only have a text-box or have a save button separately.
There is nothing wrong with that I believe. I think there is much more wrong with insisting on keeping "history" so people think that their "whoops" or "wip" are important.
All comments regarding “how you should use Git” (and many comments in general) could be prefaced with nature of the project. A predominantly markup based project worked on by a whole team of juniors will have different needs to a C codebase worked on by a single senior.
There’s not one correct way, just alternative ways that are optimal in some situations.
I can agree with the first part, because in my current job we do it. Some more than others, but it's recommended. Throwing some naturally grown git history to review is making the life of the reviewer hard. Doesn't matter whether you belong to those who make a lot of WIP commits because you are paranoid of losing some idea or whether you only commit once when everything is ready, because you are shy of showing your trial and error mess.
Probably you are also correct that most people don't invest the time to do it. That's why there is more bad code on the planet than good. It's hacked together hastily and not reviewed properly because the history is not reviewer-friendly.
I'm sure in some organizations quality work is not tolerated because "we don't have the time".
- Fix some bugs impeding the work being done. Zero or more commits.
- Refactor or lay the groundwork for the change. This could be something like plumbing dependencies, writing some feature flags, a new API, database migrations. Zero or more commits.
- Implement the feature. If you previously plumbed an API to save some text, now you can wire up the frontend. One or more commits.
- Lastly, you may have some cruft leftover that you want to remove, like an old method that you now mark as deprecated or issue a warning, or change some links or remove some feature flags. Zero or more commits.
In the end you have a series of changes that describe a cohesive item of work, or feature. It's rarely as simple as just slapping some extra markup somewhere. This makes it easy for reviewers to see the important bits and makes it easy to bisect or blame. Squashing all these changes would be a mistake.
I highly recommend folks become familiar with interactive git rebase and fixup commits. Instead of thinking linearly, and just slapping work on a branch, it should be more akin to writing a novel. Some background, set the scene, introduce the characters, some exposition, advance the plot, resolution, etc.
Keeping every branch would preserve the detailed history, unfortunately these extra branches can create their own noise.
You definitely shouldn't restrain from committing often during development, however having a cherrypickable branch history when merging instead of "let's squash this mess and forget about it" has value to me.