A note of reflection after 10 years of “A successful Git branching model”
nvie.com
nvie.com
It's so much simpler and more productive if everyone works in master, with new features behind a flag if need be, with just critical fixes ported to release branches.
There will always be arguments for why a more complicated branch strategy needs to be introduced. Resist those arguments. In my experience, the cost rarely pays for any benefits. Writing good software is hard enough already.
The only challenge we faced was maintaining patch releases. Folding in fixes for common issues which were found in newer releases sometimes involved more than just cherrypicking the fix into a patch release for an older version.
This inevitably led to asking users to update to more up-to-date versions. This further enforced the linear development paradigm.
I find the easiest way is that if you know you want (or even think you might want) a particular fix in a patch version, you can make a branch from the oldest release branch you want to support. When you create a PR, you can easily target to the release branch (or another later release branch) or master, depending on what you want. You can even do multiple PRs to merge it multiple places.
No matter how you go about it, it should always be safe to merge a release branch back into master. In fact, this is a good thing to do after you release a patch version, but can also be done whenever you need a particular fix in newer (master, or other feature branch code) code immediately.
Merging release branches like this makes following a fix significantly easier than if you do it with cherry picking. Consider if you're looking at bug ticket, and trying to figure out what branches/releases it was fixed in. If the PR was merged to master, then later cherry picked to an older release, all you'll see is it was in master. Unless someone manually comments about it or your tooling picks it up based on references, the only other thing to do is check the code itself to see if the fix is there. When you instead merge forward, even if the ticket comments are wrong or not there, you can literally follow the branches from the PR/commits and see every release it got into.
I don't find this to be overly burdensome on the tooling. If the ticket-tracking system can't automate traceability between git commits and tickets, then it isn't well-suited to git-based software development.
We use the following policy:
- A bugfix commit must fix the bug, the whole bug, and nothing but the bug. If that seems impractical for the task at hand, make it practical by breaking up the bug's ticket into smaller elements.
- Bugfix commits always cite the bug tracking system's ticket number.
- The bug tracking system observes the git tree and maintains its own linkage to it.
- When cherry-picking the bugfix to other affected branches, pass the `-x` argument to `git cherry-pick`.
That's it. The (cherry-picked from) comment helps out when just viewing the git history on its own. And the bug tracking system lets you quickly see all of the commits and pull requests that referred to the bug.
Honestly? This isn't an a problem of accidental complexity that some combination of tooling and process can fix for you. I don't think that anyone has kept the cost as low as log(N_releases). Even that would be too high for most teams.
This is well researched and discussed in Nicole Forsgren's book: Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
https://www.amazon.com/Accelerate-Software-Performing-Techno...
The software team's automated tests indicate that the software is ready to start an expensive physical testing cycle, not that its ready to go to the field.
I've worked in large companyies very successfully using TBD and releasing quarterly to enterprise customers using our software on their systems.
care to elaborate? curious about those companies and the software you are talking about.
i totally see the trunk based process work for software that doesn't need to maintain multiple versions as the original commenter said.
but for multi-versioned software you can't afford to have a single trunk and let your developers have a go at it. i went and checked the repos for some of the multi-versioned software that i use that are open-source.
well confirmed.
From a customer point of view a hotfix vs new release should be no difference. You just have to ensure that all releases are interface-compatible with the previous one. The cost of this is well worth it compared to the complexity of juggling multiple branches in parallel.
Some times they were working on changes on a release before a hotfix was addressed, and with cherry-pick's like operations, they took the changes of the hotfixes and applied to the service-pack and new versions with less effort than before they used the model.
With a dev team of 10 programmers, 20 on IT support and hundreds of client's deployments they build a profitable business (subscription-based and in-site support) where the branching model helps a lot.
They spend less time managing the releases, switching context for priority support and dealing with more changes in less time.
Trunk based development gives you that. No long lived branches. No merge hell.
It’s considerably better.
I like having explicit pull requests where a new feature can be reviewed.
But you can't open a PR against a tag v1.0.0. Hence you discover that v1.0 should be a branch, and both v1.0.0 and v1.0.1 tags. This way you can open a PR against v1.0 and easily create v1.0.1, v1.0.2, etc.
Another problem. You are at "master" and approach v13.0.0. But there are numerous bugs to fix before you release. Testing takes >24 hours. You choose to work on bugs on a branch (v13.0) this way master can proceed with new features and not be feature-frozen for days or weeks.
Another problem. Your CI is deficient in that it allows merging and cloning before it makes sure that previous merge produced good product. Puny as it is, it is the current industry standard. Thus your main branch becomes broken now and then. So you call it "develop" and only merge it to "master" what you are sure is a good product.
If these common situations don't apply to you - proceed as you are. Do the simplest thing possible, but not simpler. What you've described doesn't create any roadblock or dead end street for the project.
> But you can't open a PR against a tag v1.0.0. Hence you discover that v1.0 should be a branch, and both v1.0.0 and v1.0.1 tags. This way you can open a PR against v1.0 and easily create v1.0.1, v1.0.2, etc.
So you could just `git checkout v1.0.0; git checkout -b v1.0`, commit your hotfix and deploy v1.0.1.
> Another problem. You are at "master" and approach v13.0.0. But there are numerous bugs to fix before you release. Testing takes >24 hours. You choose to work on bugs on a branch (v13.0) this way master can proceed with new features and not be feature-frozen for days or weeks.
Or you could use topic branches for new features, and fix bugs on master at the same time.
> Another problem. Your CI is deficient in that it allows merging and cloning before it makes sure that previous merge produced good product. Puny as it is, it is the current industry standard. Thus your main branch becomes broken now and then. So you call it "develop" and only merge it to "master" what you are sure is a good product.
I don't understand your point here. Usually your CI builds a branch, runs some tests on the compiled product and makes sure the `master` branch is ok before deploying to staging or production.
> If these common situations don't apply to you - proceed as you are. Do the simplest thing possible, but not simpler. What you've described doesn't create any roadblock or dead end street for the project.
I think there are a lot of branching strategies for a reason. Your use cases can be covered with other branching strategies as well.
One could but it's often very desirable to integrate branches fast. If merging is delayed by a release there will be stale branches to merge. The experience of merging long branches scares people off of refactoring, because a refactoring creates more conflicts the longer it stays unmerged.
> I don't understand your point here. Usually your CI builds a branch, runs some tests on the compiled product and makes sure the `master` branch is ok before deploying to staging or production.
As long as you have sequential merges there is no problem. But when you merge into the main branch and it is ahead of you (due to another merge) the build may become broken. You can merge the other way first, sure. But if you want to enforce this you need support from your infrastructure.
The challenge here is that pull request might be tested again and again. It increases the CI load by some factor for large projects.
And the code review happens where? You need to:
git checkout -b v1.0
git push
ask your colleague for a code review of branch v1.0
> I don't understand your point here. Usually your CI builds a branch, runs some tests on the compiled product and makes sure the `master` branch is ok.
Gasp... You are right, you don't understand my point. You show master to everyone before making sure it isn't broken?
git checkout can take an optional tag/branch argument so
`git checkout -b v1.0 v1.0.0` works just as well for creating branch `v1.0` off of tag `v1.0.0`
1. You can implement the necessary branching with much less complexity than git-flow. Just have master, plus possibly some stable development branches created when you need it. Tag v1.0.0 off master, then make the v1.x branch and tag v1.0.1 on it. Or branch v13.x off master and tag v13.0.0 once it's stable. No "develop" branch, no long-running feature branches, nothing.
2. Importantly, you can implement the necessary branching when you need it. If you never run into these situations, you can keep having just a single master branch and not worry about it. You do not need to future-proof yourself for fear that you might run into a complicated situation. If you do, then you can make release branches at that point.
You don't need a separate "develop" / "master" branch to cover up weakness es in your CI. If your CI is good enough that you can automatically merge "develop" into "master" when tests pass, it's good enough that you can actually implement the Not Rocket Science Rule and automatically merge everything into master when tests pass. If it's not good enough, then just keep track of the last-known good commit on master, and fix build/test failures on master urgently, don't shove them off onto another branch so you can ignore then. The industry standard is the idea that "who broke the build" is a well-defined question - but it means the right step is to fix the build.
I hate "develop" and I wish I could stop pointing out how crappy idea it is.
you definitely want to get your PRs merged in to _something_ as soon as they are complete, to avoid bitrot. but just because they're complete doesn't mean you want them included in the next release.
This is the magic of git, it allows multiple merges something that wasn't possible in older systems.
That's the crux. Continous Delivery means exactly that. You want to release early, small and often. Trunk-based development and having every PR that is merged to master be deployed to production in a very short timeframe fits that very well. That's very possible with a solid CI/CD pipeline, good tests, and something that can be deployed often like a web app.
If you don't do continuous delivery, trunk-based development might be a less natural fit.
It is fine, unless you do a lot of back ports or bug fixes for older releases, at which point the version branches are very useful.
Are there any reasons at all for using git flow over trunk based development with release branches? The latter seems far simpler but just as flexible.
In a previous job, we would just cherry pick bug fix commits over from master to all the active release branches. It wasn't hard.
You can then merge that hotfix branch down into the head of master, or into any other later versions by doing a similar branch from the newer version tag first. Essentially it makes version branches lazy in that they're only created when they're actually needed, but you can still support lots of older versions to your hearts content with bugfixes, hotpatches and minor updates.
The big downside to this strategy, though, is that since you don't have a version branch until you decide you need it, new features for testing can't be merged into master until the current version is release ready (ie all outstanding bugs are fixed).
This isn't really a huge issue for smaller teams/companies or ones that can keep a fast continuous development pace, especially if your release schedule is short, because at most the merge freeze is only a couple days. But this will result in developers needing to handle merge conflicts if their feature branch exists for more than a single deployment cycle. IMO you want to keep the merge window to no more than the number of weeks between deployments in days (eg 1 week between deployments requires a merge freeze no longer than 1 day, and 3 weeks requires a merge freeze no longer than 3 days).
However, it's also really easy to grow from this into what this article describes as the team grows or the testing/deployment/merge freeze window lengthens, because you can switch from waiting to need a hotpatch for making a version branch to making a version branch every time you'd otherwise begin a merge freeze. You do have to remember to merge the commit tagged with that release back into master so that master gets all the bug fixes, but that's no more onerous than requiring hotpatches get merged into master as well.
It will work fine 99% of the time. But when you get into trouble (and a single merge done badly may cause trouble) there is no way to recover it because you lost history. And trouble is often of the form, "A month of my work went poof and nobody noticed for weeks after."
Another pain point is that the Administrator becomes a bottleneck. And when the Administrator runs into a conflict they often have no context from which to sort it out.
But on the positive side, your history will be nice and linear. Which makes tools like git-bisect very, very happy.
As others have stated, this works well when there aren't "releases" but once you have releases with versions and may have to backport things, it starts to breakdown. For how most web-based software is developed it seems to be a descent for.
It may be different if you develop something which is a library (rather than app) AND it is big enough you consider backporting changes to older versions.
If you don't, release branches (rather than tags) are nothing by a burden.
The update article mentions it was meant for versioned software.
I tried to articulate the various levels of branching you need in GitLab Flow https://docs.gitlab.com/ee/topics/gitlab_flow.html The various levels are: feature branches, production branches, environment branches, and release branches.
Devs create feature branches from master
Features will be targeted to a release at some point, Release branch gets taken from master
Devs PR feature branches to the release branch
Release branch merges back to master on deployment.
Especially on the Github repository where pushing to master is a lock and can only done by PR and merge(or squash or whatever).
Let's take a look at a release:
1. Branch of release/1.0 from develop 2. Fix some bug, then at the time of deploymennt. Merge it back to master and develop
But once we merge, github close the PR... If we attempt to create another PR for other branch. The Github now see different history between master/develop...
(Assuming master is your "next release", and develop is your main development branch)
The only way to get code into branch is by doing a PR, and click merge button on Github UI. But a PR cannot be merged twice.
And If I open two PRs two master/develop then I will see this kind of error on Github: "This branch is 1 commit behind. 1 commit ahead of develop"
If the feature is deemed good, an ‘admin’ will ‘finish’ the feature and it is merged into develop branch.
Then, when we are ready for a release, the ‘admin’ will create a release and both develop and master are updated.
Updating master triggers a webhook that deploys the release.
For a hotfix, the release step is skipped, but both develop and master are updated, again causing the webhook to trigger a deployment.
I do not use it that way, but I am open-minded to its advantages.
Our "features" take anything from 1 day to 3 weeks of effort, and generally need to roll out as a whole. They are often really upgrades of existing features, and the work to continuously merge w/ feature flags, supporting both flows/designs, is just not something we've felt was worth the extra time.
We release on a 2 week cycle, the same day each time. We merge our features into the develop branch as they are ready and safe for production.
Our scheduled release day is Wednesday. The Thursday beforehand, we branch from develop into a release/x.y.0. This builds release candidates. We have a sanity check on Friday, then our remote QA team runs through them over the weekend. Any issues they find, we try to have squared away by Monday, Tuesday at the latest.
Meanwhile other team members are still merging new things into develop. It's useful to have that release branch because it lets the rest of the team keep moving - we maybe only put out 1 or 2 developer's features each cycle, the rest are still trickling in.
Once we release, we tag, merge into master, and start over again. If any issues show up on production, we use the hotfix branch, hotfix/x.y.z. That let's us resolve issues with risking a ship of any of the changes that have come into develop that haven't had a few days of stability behind it.
For our team scale, we have a lot of users (100k's DAU), and we find this method is a nice balance between the pace of output you might get from true CI but with the reliability of a more traditional cycle.
Backend and web projects, however... Straight CI, into master, and we roll out test > staging > production when we're happy.
> Once we release, we tag, merge into master, and start over again. If any issues show up on production, we use the hotfix branch, hotfix/x.y.z. That let's us resolve issues with risking a ship of any of the changes that have come into develop that haven't had a few days of stability behind it.
In our case, there's no point in a hotfix branch because master is always considered stable. You don't merge to master unless you're ready for that code to hit production. Feature gates are used for hiding live code when necessary for marketing announcements or if the feature simply isn't ready for actual customer usage yet (sometimes we leave the endpoints live but turn off all the navigation to the endpoint so we can send out the link as a "beta" test for specific customers).
This does occasionally cause problems with feature branches lagging behind, but it's on the developer to ensure that their code merges into master smoothly (usually best done by syncing your feature branch back up to master frequently)
Another might be that git clone gets the "correct" branch by default. Might be important if your git repo gets cloned by sales, support or whatever for external demo purposes.
Anyway I don't think it makes a huge difference in complexity - it's rarely used for developers' workflows and doesn't even need to be updated that often.
So... why not branch off master and merge back into master?
This way in case of some unforeseen bugs you can fix master which is production like and then merge that fix also into develop so it is present in next release. If you would develop on master you don't have obvious "production like" place. You could do tags, but somehow it fits better with CI setup and Jenkins.
In our team we have acceptance branch instead of "release branch".
Feature branches are short lived, develop, acceptance, master are long lived branches. You also don't mix stuff because what you really want is one way direction of promoting changes develop->acceptance->master.
The fact that a change passed a CI and another change passed a CI doesn't mean they can merge and the result is guaranteed to pass CI. It will often break and if you think for a minute you will come up with really rudimentary examples of this.
The really primitive solution is to call the merge-recipient branch the "develop" and only merge to "master" when you are sure what you are merging is good. You'll be ok if you do it sequentially (no parallel merges to master).
A more complicated solution is to actually test (in CI) "what happens when I merge this" but in hiding, without showing it to others. Even more complicated is to do it in parallel. Gitlab calls it Merge Trains (Premium only). Zuul-CI calls it gate/gating and has a pretty good description here: https://zuul-ci.org/docs/zuul/discussion/gating.html#testing...
Who does QA on the feature branches, and where? Does each feature branch have its own corresponding deployment? Are they sharing databases with other branches?
When you're ready to merge, have CI test the result of the merge, and only push the merge commit if things pass. Don't let anyone other than CI push to master.
You can set up a bot with a queue to do this, of course, but you can also define "CI" as a shell script. GitHub, GitLab, etc. will automatically close a pull request if you push to master; you don't have to push the button on the website. Train people to not hit "merge" and instead to run a script, something like this which I just threw together:
#!/bin/sh -e
repo="$(mktemp -d)"
git clone https://... "$repo"
cd "$repo"
git merge "$1"
make test
git push
cd
rm -rf "$repo"
As long as GitHub does it wrong, you won't be able to "train people" not to hit the big juicy green button, that should be really a big red "Emergency Merge - Can Break The Receiving Branch" button.
That's it, that's the reason.
I am of course talking about 'outsider' changes to code, but the effect is real: the master owner can elect to say "sorry, we decided to re-architect significantly and what you did is now superfluous, or at least too hard for us to re-integrate" and almost everything I have been involved with, this is what happened.
Submitting small patches to their head state is more effective for me than trying to maintain state against a moving target with a realistic chance of making significant change, and having it accepted into the mainline.
History is for reading, not writing. Do we make decisions that give us readable and informative histories? The people who give up and rebase absolutely everything into one linear trunk are not working optimally, but at least they're trying.