Trunk-Based Development: Game Changers
trunkbaseddevelopment.com
trunkbaseddevelopment.com
"Trunk-based development" is essentially counter-marketing to Gitflow. Before Gitflow, trunk-based development was so routine it didn't need a proper name or website.
Gitflow has the "interesting" properties of being more complex than release-ready trunk-based setups, while (in most contexts) decreasing quality, and delivery speed.
Unfortunately, Gitflow is very appealing to mediocre managers who want to point to a website and some pretty pictures and say "we do that" (thus, alleviating themselves of any actual engineering or contextual thinking). So the counter-marketing is helpful, but it feels rather ridiculous how the software world is so susceptible to these trends.
It reminds me of the Alan Kay quote, "programming is a pop culture."
You can still do TBD for most of the development of new features, though.
You can still integrate to trunk as one would expect in TBD, no need for the delayed integration in Gitflow.
A common practice I see with mobile teams is they will create a 'release/x.y' branch sourced from trunk as part of their release routine (at some regular cadence, like 2-weeks).
If they need to (surgically) apply a hotfix, they can apply it to the release branch. Since the app store review process makes deploying immediate fixes impossible, a stronger emphasis on staging/RC QA is good as well.
One of the builds is then chosen as an "actual release" and promoted to the release channel in the store - the commit with the cutoff is tagged in Git.
If, for some reason, a patch needs to be applied to that exact build, you can always branch off from that tag and apply patches. But that was usually avoided and a newer, fixed, build promoted as release instead. I think we had to backpatch things less than once a year and only in very rare cases.
Care to elaborate how you decide if it is an internal / beta / preview release? Some kind of tagging strategy or are all generated at the same time?
It's added complexity and it has a cost, but there are good reasonings for it. Trunk-based and not-gitflow is fine for SaaS services or web applications I'm sure, the things that CAN be deployed in a continuous deployment fashion.
There is no notion of multiple stable releases getting patches in trunk based development.
https://trunkbaseddevelopment.com/branch-for-release/
Note in particular "the CI pipeline that guards the trunk is going to be duplicated to guard active release branches too" - branches plural.
In TBD you'd have most new development happening on trunk, but there's no reason you can't spin off a release/v1.x branch as well as a release/v2.x branch from the trunk. The point is just that these shouldn't merge back into trunk -- they're spun off forever.
You can cherry-pick patches onto release branches, but you need to figure out what that means on a case-by-case basis. There's no guarantee that the patch will apply cleanly, and there's no guarantee that it will make sense, even if it does apply cleanly. No branching strategy can fix this for you.
You have all the info you need with gitflow to go back to the state of a specific release, and then create additional releases based on that state.
Basically - there's nothing stopping you from going back to your 1.0 tag well after release 2 or 3 or even 4 are out, and then branching from there and creating a new 1.x release.
You have a wrinkle left over afterwards, since you now need to determine if any of those 1.x changes should actually make it back into develop (and this is really a case by case question) and you also (at least in my opinion) need a really valid reason for creating the extra maintenance of this wrinkle, but large paying customers can sometimes be a valid reason.
I think you have to tackle the same problem with TBD - there are certainly times when "spun off forever" is acceptable, but there are times when you genuinely want some of the work done on an old release to make it back into the mainline trunk.
---
Long story short... just prefer not to be dogmatic about these decisions and you can generally make most things work. Your customers don't give a flying fuck how you handle branches, and it should really be a consequence of what the team needs right now, rather than a "set in stone" process.
But that's also why I argue for having less process up front and creating it as needed. The master branch exists solely for the aesthetic reason of having a linear history from version to version. This is obviously incompatible with a non-linear version history.
Simplify the initial process by getting rid of master and tagging directly on the release branch.
In fact, the merge from release branch to master is a bit suspicious. I'd expect the contents of the merge to be precisely the contents of the release branch, regardless of what was on master previously. If not, what am I releasing?
It can be nice to generate release notes from the commits in the merge between the latest release and master, but you can honestly get the same info in a variety of ways (and depending on the team - it may be particularly noisy)
Basically - I think as long as tagging is happening somewhere, and that tag reflects the actual code released, I don't really have a strong preference on the branching around the process. Sort of an implementation detail that doesn't matter that much.
The typical example is a B2B company that maintains one prod deployment per customer and each customer wants a completely different release schedule
There are also several people that don't really care about release speed
If trunk has changed substantially from the release branch where the patch is required, you can't cherry-pick and you need to work on the branch.
Mind you, I've been "doing TDB or a variation of it," and advocating for it for as long as I can remember. Still, as soon as TBD was given a name and evangelists started sprawling and calling me a heretic because I favoured pragmatism in many instances, I started disliking the dogma as a whole.
I'm fine with using it as a discussion starter. Always look at principles and outcomes though. If the dogma gets in the way, discard the dogma and do what will work given your teams, their culture and their maturity.
You can either spend 4x the amount of time to make them compatible (and still follow TDB) OR spend x time and just deliver to each customer the respective feature only (and no longer follow TBD)
Also this process can continue. In the next iteration you have will 3 incompatible features with the same or different customers.
If you use different branches you can always to go to branch X and know exactly what is deployed on customer X.
In your case the only way to recreate what customer X has is to find the proper combination of libraries that you have shipped to them so far.
Imagine I am a developer and want to do a hotfix for a specific customer. I can just do git checkout branch-of-customer and start working. Isn't the process more difficult with your approach?
Having said all of this, I do think git-flow is a little complex as a general procedure. For the project I am working on now feature toggles are often better but branches (even longer lived ones) in some circumstances can be optimal.
The bottom line is some things are better in-than-out while others are better out-than-in. Let's stop being dogmatic about it.
Can we at least agree to reject the academization of this one and simply refer to all config-time optionality as a "toggle".
When a developer is happy with their feature branch, they open a pull request to develop, which gets reviewed by another developer and, eventually, merged to develop.
The code in develop gets deployed semi-hourly or on demand to a testing environment, where quality assurance takes on the feature branch.
Eventually, when the testers are happy with the feature branch, it gets merged to the main branch. Main gets deployed to production daily, or on demand. There are no release branches or tags.
Feature branches that take longer are periodically rebased on main (or main gets merged into the feature branch). On rare occasions, hotfix branches are merged directly to main (and immediately, possibly partially, deployed). That code lands in develop via an automatic merge process from main to develop.
The weakness of the model would appear to be that testing practically never happens on the main code branch. I say practically, because we do have a staging environment that reflects the main branch code version, but it's almost unused. It just hasn't been an issue in a decade+ working this way with a team of 8-20 developers. Maybe it doesn't scale to much larger teams?
Someone help me out, is there a name for this model? It doesn't appear to be gitflow, which IIUC periodically transforms (or merges) the entire develop branch into main branch (we never do this). But it also doesn't sound like it's trunk based development, which doesn't seem to have a develop branch, and also uses versioned releases. We can't be the only ones doing it this way?
1. What is the relationship between the code in the main branch and the code in the develop branch?
2. What assurance does the testing on the develop branch provide that the feature branch is suitable to merge into the main branch?
I mean, presumably you've got main and develop because they need to be different. But then, because they're different, you're never really testing the code that you actually deploy. Superficially, it seems both more complicated and less reliable than trunk-based development.
What am I missing? Why would this way of working be helpful?
> But then, because they're different, you're never really testing the code that you actually deploy.
That's true, and I alluded to it in my comment. It's conceivable that feature-1 and feature-2 are simultaneously in develop, feature-1 gets tested successfully and is merged into main; and then it turns out you needed feature-2 to actually fulfill all requirements. Seems like it'd happen all the time, but in practice, it just doesn't. Probably a combination of unit tests, backend code being written in a static language, and developers, qa, and pms using their head.
> Why would this way of working be helpful?
Having little experience with anything else, it's difficult for me to say. It seems quite flexible: we release individual features or, sometimes, related features that are compounded into a feature branch.
Main essentially never breaks, and it's nice that you can essentially deploy it whenever you want without explicitly packaging a version and testing it. That seems to be the idea behind trunk-based development, too.
But then, develop does break occasionally (i.e. a bug was discovered during testing), and it's nice that this is allowed, as well. Having to revert a feature branch from trunk because a bug was discovered sounds terrible, and also what if you released trunk in the meantime? It sounds to me like it could only work if the feature branches itself are the subject of QA (and then you have a similar problem as testing our develop branch).
> The develop branch is main + feature branches currently in qa.
Is any work done to ensure this, or is it just a byproduct of the way of working? I'm struggling to understand how it would just happen automatically.
One more question. What if instead of this dev branch people just asked the QA system to run on their feature branches? Would that generate too much work?
I'm imagining that the flow you describe is some kind of convenient or less expensive version of that.
I agree though that testing the feature branch on the develop branch is not the same thing as that feature branch living in main, because some other feature branch in develop branch may be indirectly affecting the original feature branch.
I barely use git in any meaningful way so these questions are coming purely from outsider curiosity.
(As an embedded developer, it would be bliss to experience 'develop gets deployed semi-hourly'...)
You've identified what I'd call the main weakness, but in practice it hasn't been an issue. I've discussed it in a bit more detail in a sibling post. Ymmv.
I would call it trunk based development, since feature branches are short lived.. Most MR are just 1 or 2 commits. Just part of our review process.
These stories about Smalltalk and Lisp (machines) sometimes read as if they came from a forgotten, ancient civilization that was way more advanced than ours.
When does the renaissance come?
https://www.google.com/books/edition/Mastering_ENVY_Develope...
The fun thing is the way people could subvert and ignore the recommended development process because it seemed inconvenient to them.
I always wondered if git would have been as popular if it wouldn't have come from Linus.
In your individual checkout you can have arbitrary branching structures, then it just gets squashed down to a linear model when you submit to the monorepo.
It's a bit annoying from a technical POV that there are two separate VCS, but the conceptual model works well IMO.
Personally I often have lots of messy WIPs in temporary local git branches. With some rebase, squash and occasional reflog to keep main tidy.
Frequent releases of a main branch to which developers push code (either via other branches, mr's) or direct commits... Thats what it is..
Even if you create a release branch on every release... A tag would also do.. But a tag and a branch are technically quite similar in git... Its mostly fluff
Deploy often, dont have long lived branches. Have your main branch in a deployable state 90% of the time... Hoora youre there
With a little bit of extra process (i.e. enforce that unit tests succeed on proposed merge items), you can get very close to 100% for most projects.
Probably because for most teams there isn't an issue. You only start to see the problems long term branches cause when you have multiple long-term branches in a single repo, and that only happens when your team is sufficiently big to be working on several big things at once or when your product is big enough to use a monorepo where devs on different teams work on the same parts at the same time.
If you're Google or Meta these are very obvious problems that you'll see causing issues regularly. If you're a team of 6 in a corporation or a SaaS then you'll be able to work on a branch for weeks and still be able to merge it without any big problems.
This is my main issue with TBD actually - it's a solution to a problem that few teams actually have. Throwing away a working process like gitflow when you won't see any real benefit doesn't seem like a great idea.
In the past I’ve felt a little strange that we don’t “modernize” to adopt TBD, but over doing this for many years with the same size team, it just seems unnecessary because we’re a small team.
Though those branches could be needed if you are doing mobile, desktop or library/framework development.
It's more efficient. Why prevent people from reviewing your commits and avoid merging until your feature is 100% complete? With TBD people on your team are able to review changes as they get completed and you can merge them in to avoid future confilcts and to allow others to benefit from your changes too.
Gitflow solves a problem few teams actually have. Throwing away a working (simpler) process of having a single trunk when there's no real benefit doesn't seem like a great idea.
It seems to me that Gitflow just adds a lot of ceremony of keeping multiple branches somewhat in sync through merges left and right.
I'm fine with the argument of keeping existing processes because they work, whatever they are.
Or don't acknowledge, acummulate technical debt, because due to being late CR rarely results in any deeper changes in flawed code. Then complain that project is a mess, but management doesn't want to listen about refactoring.
It is made to have few long term release versions, say you're making a database and need to support customer being major version or two behind for some years.
But if you are just making app for a client you really need just "this is released as stable and running in prod" and some flavours of dev/test environment and that doesn't need much of model to work at all, just prod/test branch with features never landing directly on prod is good enough.
Edit: I read the first hit from CircleCI and I am none the wiser. The article seems to suggest it could be any of these things:
* Having a main (`master`) branch. But literally everyone except maybe Linux developers does this.
* Not using branches? Which sounds awful.
* Pushing directly to `master` and then running CI/CD! This is insane; does anyone really do this?
So what is it exactly? Linear history?
Point three: The best results that I have had on developing systems were doing exactly that. You need more discipline (like make sure that you run test and linters locally), but my experience is that makes the whole process easier.
CI is continuous integration, there is nothing continuous to coding on your branch and merging after a few days.
That sounds error prone. Wouldn't it make more sense to have an automated system to run tests and linters for you...?
> but my experience is that makes the whole process easier.
Sounds like you would just have people breaking `master` constantly to me. And how do you do code review? Or don't you?
> there is nothing continuous to coding on your branch and merging after a few days.
Yeah I guess the meaning has changed a little (or maybe standards have changed). These days CI basically means automatically testing every commit.
I'm only seeing disadvantages compared to traditional MR/PR/diffs.
> You're just hoping
...yes, hoping that I'm not working with idiots and change the job if I would be.
It's different for OSS or any project with external contributor but you need to require some minimal level of competence
Yes, CI failing is a big issue! It means you can't bisect, and you show everyone else down while the big gets fixed.
> hoping that I'm not working with idiots
Ah you're one of those "only idiots make mistakes" types. Ok never mind; I hope you can learn that everyone makes mistakes - including you - one day, but I won't try to convince you now.
(You have successfully convinced me that trunk based development is a terrible idea though; at least for anything more than tiny teams working on small projects.)
I'm talking about people that constantly do same stupid stuff or go around system put in place coz they think they are smarter and want to shave few seconds off
We do a bunch of checks in pre-commit stuff but nothing delaying it much, and trust devs will commit working tested code. So far that social contract works just fine
And yes, you can put far less baby bumpers around stuff and go faster if your co-workers are not babies that need every sharp object hidden from them.
Pair and Ensemble programming
>So what is it exactly? Linear history?
The essence is that changes developers make are included into the trunk (main / master branch) as fast as possible. Developers shouldn't be branching off and spending a bunch of time making commits without them being added to to the main branch.
You can run CI/CD on the commits developers want to get rebased onto the trunk.
More around versioning itself, but having a single big merge is a single high risk point. Releasing everything all at once is more likely to have larger and more severe bugs than many small releases, even if you gate the deployed releases behind feature flags.
1. People stop caring about master. Then master breaks and nobody fixes it. This is a problem if not _everyone_ is on the integration branch, e.g. if there is a single small team working on a feature for the next+1 release that needs master: they can't fix all the breakage from all the other teams, and they can't get the other teams invested in not breaking master. This ruins that teams productivity, which makes development of larger scale features much more costly. Even if no team needs master, leaving master broken and coming back to it later makes those fixes more costly.
2. Regressions: people mark bugs closed as soon as it is fixed in the integration branch. Then the fix never makes it into master, and you get regressions on the next+1 release. Automatic forward merge (always immediately and automatically merge release/n to release/n+1 [...] to master) as supported by e.g. Bitbucket can help with this, at the cost of the occasional merge conflict(s) and less control in case some fixes actually should not go into a specific intermediate release.
With release integration branches you not only integrate for release (which the bigger the release, the more difficult it is), but also now you have to integrate back from release into main (because bug fixes and what not). The longer that you have had that integration branch, the more likely that you have move forward the main code and the more difficult then becomes the back integration. So you end duplicating the effort.
This sounds like a good heuristic. Can you expand on that?
Ever worked in a “spinning plates” kinda team? The kinda team who can tell you “Our release process takes 3 hours, but it’s not our fault, we’ve learned to be helpless because we feed into a dept level build and is for a good reason and and and …”. Faced with this the spinning plates mentality uses the downtime gifted by the broken process to go start up some more work in progress. Gotta keep everyone busy after all! What are you gonna do on morning stand up if you can’t rhyme off 3 or 4 inconsequential updates that none of your colleagues are paying attention to because the truth is what you worked on wasn’t important to them, just as most of their tasks weren’t important to you. Maybe there’s one dev carrying the whole team but everyone already knows what that person’s working on so stand up becomes something just like listening to the news - it can be entertaining but it doesn’t mean anything to the rest of your morning. If an asteroid decided to crash down and block the I-123 to work this morning you’d already know before you turned on the TV and so it is for standup in this kind of team.
Contrast with a team working on one thing together - they’re not accepting a 3 hour release process. They’re certainly not picking up another task just because they’re blocked. They’re swarming on resolving that messed up release process that they don’t control. You can bet that what they see as their top priority, the release process, is not management or any other responsible team’s top priority. This is really where the two approaches differ, the coherent team are now focused on figuring out how to move forward. Not picking up more work to stuff into a release pipeline that’s unduly restricted a little further downstream towards prod.
That's integration branches done properly. Linux releases are cut from that branch. Literally everything else is an integration branch and it won't get pulled (you don't get to push) by Linus Torvalds until it merges cleanly and passes all the scrutiny that he and his people apply to incoming changes.
Very few companies work like this of course. So instead you get people working on their pet branches and ignoring upstream changes. It's stupid. All you are doing is deferring integration work. Integration work scales exponentially relative tot he amount of change. So, the longer you wait, the worse it gets. The flip side is that having branches means you can distribute the integration work across those branches. And if each branch only has a small delta relative to the main branch, the collective integration work load decreases. The way you keep the delta small is by pulling upstream frequently and staying on top of the upstream changes. If you fall behind, it becomes a problem. And it's exclusively your problem.
Linux doesn't have issues with integration work on the main branch: you are either compatible with upstream and ready to merge or you won't get pulled in. It's that simple. That's why Linus Torvalds can release a stable release regular as clock work. You don't get to stamp your feet and whine about how important your branch is. You do the integration work before your changes are pulled in and it's exclusively your problem to make that happen. So people, integrate upstream changes often and Linus keeps unstable changes out of upstream. They front load all the integration work before anyone even talks to Linus Torvalds. Done right, this scales to an ecosystem with thousands of contributors from a multitude of companies across the globe. It works by distributing the integration work across the ecosystem.
This can work in large companies too. But you have to do it properly and with intent. Pull requests are pretty popular for a good reason: they work the same way. If you have to have long lived branches, somebody needs to worry about those staying in sync with upstream. If that doesn't happen, you have a problem waiting to happen.
Release branches have a place where you are releasing software and supporting multiple different release versions for different clients, because you need to be able to bug-fix older versions and merge fixes forward rather than just fix current and re-release into your own system.
If you only ever give a shit about ‘latest’ then you don’t need to bother with them. IMHO.
It would be very hard for a large company to adopt their monorepo because you'd have to replicate their tools.
Just in case anyone was thinking of just copying Google...
So the feature/bug/etc. branches can contain multiple commits which are helpful for the developer(s) that's working on it but it all should be squashed into a single one during merging into master.
In short the history noise in master should be kept in check and reduced as much as possible which makes e.g. bisect easier as we don't take into account pointless commits and the history is cleaner. Problem is many think "more is better" and they end up with branches that are much more complicated to work with and reason about.
I have found benefits though to structuring your WIP in a way that it can be merged into the trunk with minimal risk of breaking existing stuff. This is a mindset change around favoring adding new modules / components / endpoints without clobbering working code, especially for larger projects where the code is going to come in over the course of many releases.
Maintenance releases are done by going back to the tag or branch for your current prod release, making a new release branch from it, and pulling in whichever commits are needed.
But hey, I am not arguing for feature flags. No, Sir, please no.
Why do you prefer not to work with feature flags? For context, the product I am working on (which is not very big) uses a simple development/production split to enable or disable features. Several people have recommended using feature flags, but since I have not previously worked on projects that utilized them, I want to learn more about the advantages and disadvantages from a developer's perspective.
Of course, a branched code base in the limit is similarly never merged which obviously is terrible as well. My main point is don't follow trends, instead simply do what what makes sense - that might mean feature toggles, branches or some combination. Stay awake, be creative and avoid dogmatism: sometimes things are better out than in, other times better in than out.
Are you saying that people in this thought experiment would go so crazy with feature toggles that they'd develop several different versions of the same feature, each with different toggles?
The most obvious being, it's a lot easier to see everything when it's in one place.
That, to me, is the difference between committing on a branch and merging back to trunk/main. A branch is just a way to record and share an idea. Merging back is saying with some confidence "this is something we should build on". Feature flags allow you to disentangle that social signal of "we should build on this" from "this works in production".
I'm not sure (and curious) about other ways of working that can fulfill the same collaborative social signaling function.
Advocates of methodology will often speak in absolute terms. Plus their way of doing thinks is sometimes presented as a big magical revelation. This article is actually not like that, it reads like a historical summary.
It’s of course good to sometimes think about other ways of doing things. But in the end the way you’re using version control should reflect your requirements and development practice and not the other way around.
We do 1 main branch plus tagging. That’s it. It’s simpler for us and it’s a motivator to write code that works _now_.
But if we ended up in feature flag hell we would probably reconsider or simply branch off. Just be pragmatic about it.
There are of course companies with predefined rules about these things. I can’t speak to that.
- a merge process that happens on CI
- and that takes some time, running checks on the merge branch with main merged into it to effectively test what main will be
- and you want to make sure that there's no contention, so you finish one merge before testing the next
then you effectively have a merge lock in CI.
And that can lead to merge convoys where lots of merges to main line up and take a long time.
Google's trunk-based monorepo approach is a developer-centric approach seeming primarily driven by dependency management concerns.
After having used release and feature branches, and trunk based development, as a developer I see mostly benefits (there are some drawbacks, there is no perfect technique) with trunk based development.
Edit:it->them