Stop cherry-picking, start merging, Part 1: The merge conflict
blogs.msdn.microsoft.com
blogs.msdn.microsoft.com
I will sometimes make a fix in a feature branch that should go onto master before the feature is ready, and cherry pick it to a new bugfix branch. Once it's merged, if you rebase the feature onto master git will normally sort it all out, or you can use rebase -i to skip the cherry picked commit manually.
If that is not the case and you have previously pushed your branch somewhere remote already, rebasing would end up rewriting history and things get a bit more tricky. You can still get away with this if it is just you.
Now here comes the crux: if you have a branch that you've been working on for several days, you really should be pushing it regularly to remote so that you don't lose your work. Also you should be creating lots of small commits and create a giant commit that touches everything. So if you then go and rebase, you are potentially rewriting history for someone. If that is at all going to be an issue, the better solution is to create a new branch and push that (or for Github folk, create a new pull request).
Many bigger projects will be configured to flat out reject rebased histories overwriting existing ones. Most of the public projects only allow you to create pull requests that have to be accepted by someone with commit rights on the main repo. And many of those people prefer clean rebased histories when they are merging so that they can do a so-called fast forward merge where the history stays nice and linear. If you have a branch with lots of back and forward merges, that is likely not going to go down well. It makes reviewing a lot harder and you may be better off creating a new branch and then rebasing the old one on top. Typically you'll have a bit of work re-solving all the conflicts you resolved earlier. This is a good argument against long lived branches.
Cherry-picking is sometimes useful but it potentially creates issues for the subsequent merge (fast forward or not). The nice thing with the pull request model is that it is not the problem of whomever cherry-picked but only that of whomever created the branch. If there is a conflict, it would be theirs to sort out before they can get their PR merged. If you are cherry-picking small fixes, this is usually not a big issue and a perfectly valid way of solving a small issue without landing a lot of work in progress.
I do this at work for all the feature projects I work on with my team.
You mostly run into problems when people get confused about what is happening when they see the branch history has changed.
Git histories are wonderful and immutable so messing around with commits is quite easy if you know how to move branches around to undo changes if you mess up.
It's a feature that gets turned off server side in bigger projects and on OSS projects you don't get to push your changes at all and instead you need to work to get people to pull your changes. I'd argue that's a great way to isolate yourself from people misbehaving on their own branches since it is entirely their problem to get their branch in a a shape where it is straight forward to pull (as in fast forward merge). Rebasing can be a valid tool to get there. Force push is not thing generally in such projects except maybe on your own private fork.
If you need to move a fix into multiple branches, isolate that fix in its own branch to begin with. The entire branch is the cherry. Each target branch deserves its own merge because chances are, seeing as the thing is so critical, the merge needs adequate attention each time to get done correctly (causing more work and friction is sometimes healthy).
If you're reaching the end of your sprint and your branch isn't ready to merge, chances are the entire branch could use another sprint in the oven. You've completely run out of time and you want to isolate a few changes, that's great if you want to move fast and break things.
In TFVC (and SVN, CVS, Perforce, etc.), branches are expensive. It has taken me months to "unteach" people at work antipatterns that arise due to that. Management still doesn't get it - there is this holdout perspective that branches have to be micromanaged. This was the only sensible way to manage the mess caused by TFVC+co.
If you're finding unusual degrees of friction with Git (Mercurial, Bitkeeper, etc.), chances are that you're artificially introducing issues caused by your legacy version control and trying to solve them. Start forgetting and unlearning.
http://svnbook.red-bean.com/en/1.8/svn.branchmerge.using.htm...
Apart from sarcasm, SVN branches use suffer-on-merge: creating a branch means immediate technical debt of the most useless kind, which is more expensive than any inefficient file copying.
From the same manual:
"To perform a sync merge, first make sure your working copy of the branch is “clean”—that it has no local modifications reported by svn status."
"One special kind of flexibility is the ability to have a working copy containing files and directories with a mix of different working revision numbers. Subversion working copies do not always correspond to any single revision in the repository; they may contain files from several different revisions."
If me and Jim are working on two features on the same release and then I find that I need to integrate Jim's work in order to continue mine, Jim will need to merge his [incomplete] work into a common ancestor. If I am working on a feature in a completely different release, this "merge path" gets long and begins to cost money and carry risk. Jim hasn't finished his work yet; before any release can happen, Jim has to finish his work and merge the complete changes to all affected branches. If we can't wait for him to finish, then we're rolling back his changes and, oops, someone rolled back the same change on multiple branches.
Now we can't launch a hundred features because there was an interdependency between two.
At work. we regularly merge upstream FreeBSD into our codebase, and run a suite of tests to make sure that we have not introduced a regression in performance or functionality. These tests take days to run, and roughly another day to analyze.
What causes a cherry pick is if we realize late in our release process that we're suffering from a bug that was fixed upstream (or that we can fix upstream). Since we're late in the release process, we don't want to take a full sync to the tip of the upstream master and consume potentially unstable changes, as that will mean re-running all our tests and resetting the clock on the release. So we cherry pick the fix (sometimes after committing it upstream ourselves).
Not sure why, but I always knew from the beginning that cherry-picking is something not to be done from a live in-development feature branch without taking steps afterwards.
Cherry-picking is important in the case that you mention, master to release, the opposite direction could be a merge, ie release branch to master.
https://blogs.msdn.microsoft.com/oldnewthing/20180314-00/?p=98235
http://wiki.monotone.ca/DaggyFixes/https://medium.com/@kurtisnusbaum/stacked-diffs-keeping-phab...
I've been using stacked diffs for years now after trying various other git workflows, and I love the freedom it gives me. I almost always structure my work as a single branch (not a separate branch per feature), one commit per code review, and I'm free to edit, reorder, split, and combine changes as I please (mostly with interactive rebase). When a commit gets accepted, I just move it to the bottom of my stack (if it's not there already) and push it. It makes it easy to split changes into a "pipeline" of logical steps that can each be independently reviewed and landed, with multiple steps in code review at once.
I hate it when people treat dependent code reviews as an "advanced feature". They're only advanced if you make your workflow so advanced that they're hard to keep track of (many commits per review, many branches, merge commits). If you keep it simple (one commit per review, always rebase), that simplicity gives you much more power in other areas that IMO are more important.
I just wish it wasn't so much of a pain to get working in GitHub.
My main point is that merges are not easily computable. comm3 = merge(comm1, comm2) is not a kind of function you can run and see for yourself. Instead it is a kind of hand wavy magic in which we declare that those two apples plus a lemon equals three mangoes. You throw the beauty of hash tree out of the window like an overdue christmas tree.
Much less this handwavy commit exposed to uninterested parties.
Although you still fix the conflicts each time, it means you don't have many merge commits as you go along
Most people do that though because they don't want potentially unbuildable commits on master. And the value of keeping around WIP commits from a feature branch is questionable.
The major downside of rebase is that even if you don't squash/skip it changes the hash of every commit. This is very problematic when others have ever checked out your branch locally, or made commits that haven't been pushed. It takes greater communication between the team in my experience.
You can also, as a substitute for greater communication, establish a protocol around branch naming and follow it.
What works for us is using the word "release" and "wip" in our branch names. If a branch is a "release" branch, then it is safe to base other work on it. If a branch is a "wip" branch, then it is not safe for merging upstream.
These two are not exclusive. For example, you might have a branch "dev" and a branch "release", and maybe another branch "release-dev-wip" – these all have different properties. "release-dev-wip" is a short-lived merge target for features that might not be completed yet. It is not safe to merge this upstream, unless you've checked with all of your colleagues who merged feature branches to it, and they all certified that it is no longer "Work In Progress."
The key I think to make this protocol work is to distinguish between "feature branches" and "environment branches" – dev is an environment branch, and a permanent one, so it should not be rebased. Feature branches merge back to environment branches, and environment branches are deployable.
You can break this rule, say if someone hotfixes master, which is upstream from dev... but it should probably be an exception to do this, and not a regular occurrence, as many people may have already based their work on "dev" and they will all need to rebase on the new (rebased) dev, in order to get a clean merge later. This is where the communication is not always optional. It might be a better choice, if the hotfix is unavoidable, but the project is large and this type of communication is logistically impossible, to merge in reverse (checkout dev, then merge master – back to dev). It might be ugly, but it's considerate. Either way, there should be a clear protocol and no ambiguity on the matter of whether you have a feature branch or an environment branch when you hold it in your hand. Feature branches represent work, environment branches probably ought to just combine the work, and maybe keep a record of how it was deployed.
The branch "dev-wip" is a temporary environment branch – it might be deployed to a dev environment for example, but you should not expect it to remain permanently in the git history. At some point, perhaps it will be renamed to describe the features it contains, and then rebased and merged back to dev. If you merged your feature to it, you might expect that you will need to keep the feature branch around, so you can rebase it on "dev" or "master" later, and finally merge it back.
The whole branch might not get merged upstream at once. (You can also call it "release-dev-wip" and then, the person who looks at it will know that it may contain some completed features that for some reason were not ready to merge upstream, but perhaps should not be discarded entirely. I personally like to rebase wip branches on their upstream before discarding them, just to be sure I'm not throwing away someone's work that they may have thought they merged.)
Protocol is just a different form of communication that is done up-front. If you decide on a protocol and forget to explain it to your team before you implement it, you will obviously not have solved any problems. It's also important to be clear and confirm understanding, so that you can be sure nobody is imputing meanings that you didn't intend. Some teams might choose to only do prod deploys from the "release" branch, and that anything in the "master" branch must be safe to merge to release and send off to production. You could easily get yourself into trouble if you didn't understand when your team expects to work this way. Some teams might prefer to organize their releases on a "release" branch, and then use Continuous Delivery to trigger prod deploys when the release is merged to master. Other teams might prefer to use a tag for that.
Mostly I think we can all agree that you should not rewrite a commit once it has been tagged, but again, this is not something that is strongly enforced by git, so it may vary from team to team. If a release that was tagged broke prod, it might actually make sense to wipe that tag from history and reroute the master branch around it. I've never seen that, but I think you're right, the most important thing is to communicate with your team so there is no ambiguity around these kinds of expectations.
Being able to completely rewrite a feature branch at will before merging it (or rebasing or cherry picking or whatever) to master lets you go full on XCKD[1] with your commit messages but still compress them down to 1-2 logical "add change 1" "fix bug 2" commits at the end.
Thinking about commit messages mid-flow isn't worth it ;)
Yeah, I hate rebase, but the git boys (a very clever bunch) seem to like it.
There are three ways to rebase that you should learn, and I only have 5 minutes, but I'll try to sum them up quickly:
rebase -i: This takes a single upstream (parent) commit as an argument and assumes you want to change something, or everything after that. That was the most important part to understand... rebase assumes you have some stable base, and that is the commit that you want to keep. (You can still squash other commits into it, or rewrite the message, but if you want to remove it altogether, you should pick the commit before it as the stable base.)
Say you have a commit where you tried something, and then three commits that follow which shored that up into a working something. You may want to use rebase -i and squash those small commits, to make one good commit. (You might also use "fixup" to erase those commit messages about fixing something, because this time you're just going to deliver one commit that works. As a reviewer, I often tell my coworkers that while history is nice, I don't need to see every little thing you tried if it turned out that simply didn't work. If you want me to review your commit history, you should review it too, and feel free to clean it up first before sending it over to me!)
git rebase [original-base] --onto [new-base]
This is like cherry-pick for branches. This is where `--preserve-merges` really shines. Say you have a branch that you merged another branch into, and you merged that branch into your wip-release branch, and it still isn't finished, so you want to preserve all of that history when you rebase it ahead of master, where someone else has already merged their changes in before you. The tool for this is rebase --onto. Imagine you pruned the tree, leaving the node where your work started "original-base" intact, and grafted the branch containing your work that you trimmed onto "new-base". This almost has the argument semantics of a tool like "mv" -- mv [from-file] [to-file].
Then last but not least, git pull --rebase
Someone has merged into your upstream (say, master) ahead of you, and you don't have any interesting merge history that you want to preserve. You just want to pretend you started your work after they finished theirs, and any empty merge commits you might have accumulated since then can be discarded, because it's really just one feature branch you're working on. (When you get good at working this way, you might actually find that you merge more often simply because it's so easy to discard the merges when they are no longer needed.)
You may want to do this sooner than later, because you suspect their will be a merge conflict with their work, and you want to resolve it now, while their work and their merge is fresh in their head... so you can ask them what they were thinking, and this merge won't be looming over you for the duration of your feature branch. Depending on how you work, feature branches or wip-release branches could live for a duration of several weeks or longer. Pull rebase works like `git pull` in how it resolves merges, except it takes your commits and applies them one at a time, rewriting each commit to resolve the merge. This strategy won't generate any new merge commits.
OK, one more! This one isn't a rebase, but you'll probably want it if you are using all of the above... "merge --no-ff" – once you are an expert at rebasing, you are probably going to miss those clean merge commits, because by default all of your merges are going to be fast-forwards, and there's actually some merit to the idea of having a merge commit that shows "here, a branch was ended." Basically every Github or Bitbucket will do this for you when you merge a Pull Request without choosing the "squash" option. It's handy to have in your command-line toolbox as well. Your co-workers will know that they can base their work off of yours and they can trust that you're probably not going to squash your commits behind that point again, but you can get all of that without the overhead of opening a pull request, and putting it through a formal review.
This does not have the ugly semantics of a regular merge with two parents (that is the merge that everyone complains about, which you maybe cannot readily express as one "diff" against a single parent commit). As long as you use it together with rebase you can have a clear merge history while always maintaining the capability of tracing each commit's lineage to one parent with a single, clear diff. You just get a nice empty merge commit to show that "a merge happened here."
There are many ways to use rebase, and not all of them remove history. Hope this helps!
I did not know about rebase-merges!
> The --rebase-merges mode is similar in spirit to --preserve-merges, but in contrast to that option works well in interactive rebases: commits can be reordered, inserted and dropped at will.
It's always possible to make smaller commits, and it's always possible to cherry pick onto master regularly. If you're not doing so, you're being lazy.
This style has more benefits than easier merges. Smaller commits are easier to test and less prone to bugs. Also less likely to be affected by context switching.
Maybe our stories are too big, but a feature branch in our company lives at least 3-4 weeks with 4 developers working on it.
The only time I use rebase is when I got local commits, and I want to pull changes from my team mates also working on the same branch.
If you cherry pick to master regularly, rebases on master are never cumbersome.
Surely the whole point of a long-lived branch is to contain stuff that's not in master - ie things that are being deliberately kept out of it?
The point is that you merge anything that is safe to merge, which is usually most of the code if you find a way to hide it or otherwise incrementally change things.
I like small, atomic commits that do just one thing. I frequently rebase onto master whenever I'm working on a feature. When my branch is good to be taken into master, I do a fast-forward merge (basically a commit by commit patch application); no magical merge commit from nowhere. Of course, FF merges are possible only when you're rebased.
Things people not used to this workflow might think will be a problem:
>Conflicts when I rebase from master
Solve them. They're going to happen anyway when you merge into master, so it's easier for everyone if you resolve your conflicts then and there.
>Multiple people need to work on the same feature branch
Sure, do your work locally. Before pushing, fetch the remote branch and rebase onto it as well. Resolve conflicts and your work history will be pristine.
Ask everyone on the team to do the same and nobody will have to resolve somebody else's merge conflicts (because you're the best person to resolve your conflict).
I don't know what's FF merge, what's no-FF and what you do when your FF merge turns into no-FF.
Almost all branches have related history, it is very rare to have completely unrelated branches with separate starting points.
Most git users in touch with the design goals of git use branches sparingly, either for long lived projects (with hundreds of commits) or as a sandbox in which work is frequently rebased into a concise history of a few patches, which are usually integrated upstream without a merge. Merges are also used to merge disparate trees, like when Linus merges in trees from around the kernel development scene (since each kernel subsystem maintains their kernel tree as almost independent projects, using merge makes sense to integrate the foreign histories into the upstream kernel). The Linux kernel approach probably requires a full-length article to fully explain, though, so don't read too much into that.
Are the only downsides that 1) users don't understand how kernel development is done, which extends far beyond source control; and 2) a merge commit is left as a marker?
The downsides include the creation of a merge commit, sure, but also the fact that it's used as a crutch by users who would rather not strive to understand how git works.
>You should only use merging to merge two divergent or unrelated histories.
It's better to merge often and rarely if ever allow two branches to even have "divergent or unrelated histories".
>It's better to merge often and rarely if ever allow two branches to even have "divergent or unrelated histories".
Remember that git is distributed, not just nominally but also in workflow.
I can think of only one well founded case for wanting to rewrite history (instead of adding metadata to the history, such as merge commits and revert commits), and that is polisihng the history after the fact to hide what happened in the sausage factory. This may be a legit reason if you are not working in a safe environment or otherwise want to prepare for a future adverserial setting with some people who will see the history. A special case of this is when you accidentally commit a secret into the code, but that case has satisfactory technical recipes already.
If we want to improve the s/n in the log messages, a good avenue is better commit messages and metadata for the merges. For example in a github PR based workflow the main change note in each unit of work is the PR's description, and the individual commit messages are just minor notes along the way.
(While I would hope the answers to this question would be widely useful, I mostly work on C# applications, with less focus on JavaScript-based applications. Maybe that's relevant for deciding on strategy - hopefully not!)
Personally, I recommend the GitHubFlow and CI/CD for "agile", and never branching per-environment but there are almost as many opinions as there are developers.