Trunk Based Development | Hacker News Reader

Trunk Based Development

trunkbaseddevelopment.com

I've rarely seen the first description of it where people actually commit directly to main. Except in very early stage projects. But it does always feel the fastest if you only review code "on-demand" in PRs/MRs instead of enforcing it for every change.

I think in a team with good ownership, enforcing formal reviews slows down a lot. But of course in a larger code base where no single engineer can understand all the effects a change might have, having a bit of knowledge sharing and 4 eyes enforced is often the better approach than yolo-ing it.

Then again I did build an SQL review tool for database access because I felt like "yolo-ing" production access was also not the way it should be. It's an interesting slider between full autonomy and strict review processes where each team needs to find their sweet spot: https://github.com/kviklet/kviklet/

We're organized in small teams around specific products. That leads to mob programming being a good fit, so code reviews and knowledge sharing is organic. This gives us an opportunity to commit to main and do direct deploys to production. Treating every commit as deployable is key, but it drives good practices.

As I understand it, trunk based development does not call for committing directly to main. It says to avoid long-lived branches for releases, whole features, etc.

There's nothing wrong with small, short-lived branches that can be quickly reviewed and merged into main.

That being said, I've been in a small team where the blessed style was to commit directly to main and do reviews "on demand". It quickly gets features deployed in a way that builds a lot of rot and debt into your project that people quickly lose a good understanding of. Then you just keep piling.

There's probably a way to get this done properly with a tiny team of very experienced and well-aligned developers; but that's usually not what you have in an environment that pushes this kind of extreme no-review-interpretation of trunk-based development.

Slow down, do reviews, avoid keeping branches open for more than a day.

> well-aligned developers

I think this is very key, if the development style and the direction of the project is clear, much less review and alignment is necessary.

And also

> avoid keeping branches open for more than a day

Big +1 on that, fast reviews are extremely key. Most teams I have seen often took days or even weeks to merge branches though, often because you end up waiting too long for reviews in the first place. Or because of good old bike-shedding. But also because these code reviews often uncovered uncertainties that needed longer discussions.

However usually code is easy to change, so defaulting to "just merge it" and creating followup tasks is often the cheaper approach than infinite review cycles.

I think it's still worth-while to do reviews. A second pair of eyes does wonders, and it spreads knowledge of what things exist and how they work. If changes are small, reviews can be quick. It's possible to keep building on top of code being reviewed, and even easy when using modern VCS tooling like jujutsu.

Once the code is merged, chances are it will not get changed Those follow-up tasks will be displaced by more pressing work that will keep piling onto a slightly unstable foundation, increasing the tilt over time.

There is an excluded middle between "no reviews" and "infinite review cycles": proper, timely and efficient reviews. They are worth investing the time to get right. They will start paying dividends months down the line, and boy will they keep paying.

This is not about trying to get things perfect from the get go, but to get them done right while you're there. "We'll fix it later" is not gonna happen, and is much more expensive than it initially seems.

> However usually code is easy to change, so defaulting to "just merge it" and creating followup tasks is often the cheaper approach than infinite review cycles.

I wish this was the "default" mindset everywhere, especially in those cases where you have that one colleague that loves to nitpick everything and doesn't see an issue with holding up both releases and wasting your time over menial pedantic stuff. It would be so much easier to merge working code and let it work, and keep those TODOs in the backlog (e.g. trash).

In a sane world, code review would be like:

  1. Will this work and not break anything? We checked it, it's okay. There are no apparent critical or serious issues here.
  2. Here's a list of stuff that you can change if you wish, here's why it might be an improvement.
  3. Okay, we have some left-over nice to haves, let's keep track of those for later (or not) and merge.

It gets infinitely worse if you're working on 3 projects in parallel and the person wants long winded calls or starts nitpicking about naming, or wants things done exactly their way as if it's the only way (doubly worse if their way is actually worse by most metrics and tastes).

> "Scaled Trunk-Based Development"

> There's nothing wrong with small, short-lived branches that can be quickly reviewed and merged into main.

I would have called this "branch based development", personally.

Hmm, branches you say? Branches that contain features?

https://www.atlassian.com/git/tutorials/comparing-workflows/...

But yeah, that’s what people generally settled on, naming wise.

What counts as "long-lived"? Our shortest lived branches are quick fixes because something got merged into main that doesn't work properly.

I'd like main to always be ready for production, but that seems an elusive goal no matter what git workflow you use.

The best way to prevent complex merges does not depend on your git strategy, but on how modular you make your code. If a change requires changes to only a single file, and your files aren't too big, there's little chance of conflict. The more files need to be changed (often because the same thing needs to be declared in 4 different places), the bigger the chance of conflict. Same with larger files. Each file should have a single concern.

For the purposes of this discussion, I'd say a "long-lived" branch is one that keeps running into merge conflicts with other code that's been developed in ignorance of the branch's contents, or a branch that acquires enough changes to not be easily reviewable in its entirety. But generally, I'd say if it lives longer than a day, that's getting too old. (Exceptions apply.)

Code should live in main/master and be in a production environment as soon as possible, but it does not have to run for all, most or even any customers/clients from the start. You can gate it behind feature switches, preprocessor directives, global constants, whatever. As long as it has an active purpose and is being developed further, it is not dead. Ideally, it will have tests.

The result of merging back to main often is that other changes will be made in awareness of your new code, and all merging will happen organically and in small increments. You also get to develop complex features bit by bit, in a way that can be verified in a real environment in a controlled way. And everything can be deployed anytime, which should happen often.

Being able to verify incrementally and get quick feedback lowers the risk of otherwise releasing a hitherto completely untested large feature into the wild in one go.

This does not absolve you from making sure that your changes actually work as intended. That will be easier if you portion the work into smaller chunks that are easier to understand than a whole, complex, all-or-nothing feature.

Of course you will still have to make sure that your small chunks fit together and make up a good "whole". However, the idea is that you're doing this exact work anyway when developing a larger feature on the side, but you might as well cut it up to deliver in much smaller increments.

This approach has its own costs and drawbacks, but in my experience it often produces a substantial net-reduction in friction and accumulated complexity and risk, and it keeps things moving, which is a benefit on its own. This is from my perspective in backend development, and it will not apply universally, but widely. Essentially, whenever you can get away with it, you should strive to deliver like this.

In my previous job we worked like this. We had one dev branch that everyone pushed directly to and the dev branch was eventually branched to an RC branch which in turn was merged to master once the release was complete.

The team was small, around 6 people, and the codebase was maybe medium sized (~500k LOC). There was no formal review process, instead it was up to each team member to ensure the quality of their own and others code. In practice I would read through all commits that came in the previous day while having my morning coffee. If there was some egregious I would talk to whoever made to commit to make discuss if something should change, but this was fairly rare.

Formal PR reviews were only ever really used for new members or for bigger/sketchy changes where someone wanted more eyes on it.

Because I ended up reading most commits, I ended up knowing how pretty much the entire codebase worked. It takes a while for this to develop, but the more you do it the better you get at it, especially in the context of a single codebase.

I work in games, and we do commit directly to main. On a smaller team you can get away with pre submit review, post submit checks. On a bigger team you need pre submit checks but honestly the point where you need this is much much later than you think. One of my previous projects had 100+ people committing directly to main with no pre submit checks and jt broke once or twice a day. The builds took longer than that to go through so you just always sync to “last known good”

Working without branches, except for releases, is the most effective way of working, using rebase instead of merge to get a single line of commits. Even release branches can be avoided with continuous deployment.

> Working without branches, except for releases, is the most effective way of working, using rebase instead of merge to get a single line of commits.

I think you're confusing workflows with commit history.

You can work with feature branches all you want, rebase them as you feel like it, and then do squash merges to main.

The likes of GitHub even have a button for this.

Why in the world would you do squash merges? ...except to clean up messy mini-branches written by total noobs. I don't do separate commits for funzies. If you want separate commits for ease of review, why not for later reading of the code.

Assumption: above mentioned total noobs don't use git rebase -i or equivalent, everyone else does

It’s pretty hard to keep the commits in a working branch in a good legible state - certainly it takes work to do it.

In 25 years of professional development I’ve never really had a situation where the commits on a branch would have helped me understand what was going on a year ago when the work was done. That includes pretty big bits of project work.

I’d much rather have a trunk with commits at the granularity of features.

100%. I don't want to know how the sausage was made. It's similar to research papers, or history books, where the way we arrive at results or outcomes in the real world is often quite different from the way it's presented in the final form.

A good commit history is more like a well-written sausage recipe than like a TV documentary about scandalous sanitary conditions at Foo sausage factory ;)

`git log --merges --first-parent` gives you both.

I've had separate commits come in handy several times when `git blame`ing when working with people who actually described what changes were about in their commits (which, unlike comments, don't go out of date).

In 25 years of professional development I have several counter examples where some bit was either a trivial git revert of a single commit - among multiple ones in a branch - away, or an absolute pain because the squash-merge commit had flattened too many concerns together, concerns that were perfectly split in the topic branch but that branch was long gone by virtue of being auto-deleted on PR merge.

Coincidentally, every single squash-merge commit advocate I've had the unfortunate debate with was a regular practitioner of public tmp / tmp / try again / linter / tmp / fix / fix / haaaaaands commits.

Note that I'm not against squashing/history rewriting e.g rebase -i and stuff (which I'm a heavy user of so as to present sensible code aggregation reviewable per-commits), only squash-merge.

If you work with a ticket system, squash-merge gives you the same granularity, where a commit would refer to a single ticket.

A ticket should be atomic describing a single change request. PR in this case are the working room. It can be as messy or as clean as you want. But the goal is to produce a patch that introduces one change. Because if you would rebase -i at the end, you would have a single commit too in the PR.

No, you wouldn't. git rebase -i is to remove noise, which is about merging commits that, well, make more sense together than apart. Which is mostly about summarizing trivialities (e.g. several typo fixes) and squashing fixups into commits that introduced a problem in the same branch.

A typical bugfix branch might look like this after rebase -i:

Move property to a more appropriate place

Improve documentation of feature Foo

Fix accidental O(n^2) in feature Bar

Fix interaction of Foo with Bar

Those looks more like noise to me. A squashed merge (or a final squash before PR) would be:

  TN 43 - Fix mismatched interface between Foo and Bar

  We've moved the X property to a more appropriate place and
  improved the documentation for Feature Foo. We've also found and fix
  an O(n^2) implementation in feature Bar.

The the ticket TN-43 will have all the details that have lead to the PR being made: Bug reports, investigations, alternative solutions,...

The commit message is what's more important. I don't think I've ever needed what is in a merged branch. But I've always wanted the commit at one point to have tests passing and a good description of the patch. And all the talk in the engineering team are always about ticket. It does makes sense to align those.

They aren't noise at all and have found them useful a bunch in the past when I worked at a place that didn't squash. Commits at this level act as immutable comments that don't get out of date. Provided you do --no-fast-forward merges, the merge commit is the feature commit and you can get the "clean" feature history with `git log --merges --first-parent`. Best of both worlds! Being able to `git blame` and get a granular message about why something was done can be really handy, especially when looking unfamiliar code.

I get where you came from, but I prefer having a more holistic view of a change, especially from a product perspective. So even when git-blaming, either I’m reading the current file or I go straight to the log of the commit (with message and diff).

I prefer granularity at a product or team level decision. Not workflow details.

I'm not trying to convince you to adopt or anything, but I'm saying you can have all of that without squashing with the caveat that you would need an alias to jump to the merge commit. Otherwise, you just treat merge commits as you would a squash one. Merge commits are just like regular commits that can have a custom message and show a diff.

> If you work with a ticket system, squash-merge gives you the same granularity, where a commit would refer to a single ticket.

With GitHub you can squash any PR merge. The link to the PR will include the complete history of the feature branch prior to the merge. Even the commit history prior to force pushes is tracked.

I take it you haven't had the pleasure of working with your average ("dark matter" as they're called here) developers. I wouldn't call myself an "advocate" of squashes, but it's often the only practical way of keeping git history somewhat usable when working with people who refuse to learn their VCS properly.

I chunk my changes into tiny commits ("linter"/"tmp"/"wip"), but then rebase aggressively, turning it into a set of logical changes with well-formed commit messages. git bisect/revert work great with history written in this way even years layer.

But: most of the people I've been interacting with also produce lots of "wip"/"tmp", but then skip the rebase. I can only offer my help with learning git rebase for so long before it starts taking too much time from the actual work. So squash it is: at least it produces coherent history without adding thousands of commits into `--ignore-revs-file`.

And sometimes, a patch is just that big. especially in UI works where a single change can cascade down to multiple layers.

> I chunk my changes into tiny commits ("linter"/"tmp"/"wip"), but then rebase aggressively, turning it into a set of logical changes with well-formed commit messages. git bisect/revert work great with history written in this way even years layer.

In a PR based workflow, it has become easier to have the PR be a logical unit than to `rebase -i` all the time on my end.

I on the other hand have never come across a scenario where I run git bisect to find a commit that broke something, discover a small commit as a culprit and wish I had instead found a commit that's hundreds of lines long.

What has happened a whole lot though is the exact opposite.

It might be better to view a commit as a natural unit of working code. There are a lot of units of working code which would be tedious to be introduced as a only a few lines.

As such, a new codebase is likely to grow by large unwieldy commits and a mature one by targetted small commits.

Our strategy is to squash on merge and ensure the JIRA ticket reference is in the MR title. You have the granularity of the feature which is going to help guide you on the intention. It's also much easier to enforce. People like to write and commit code in their own way.

For me, all the code in the PR at that point is in question. The code was changed as part of a bigger part of work and reverting it without understanding why might cause issues back the other way.

The other thing to say - that is missing from this conversation - is that if you’re using github, the original commits are still against the PR.

Each commit should be small, have a descriptive commit message and be stand alone. I consider the Linux kernel a good example of how to do commits and messages right. Often the commit message is longer than the code change.

I strive to do that when making commits for work too, and that helps when going back in history and looking at history to motivate why a change was made.

While working I rebase all the time to move changes into the relevant commit, I don't find that particularly hard or time consuming. Doing this upfront is easy, splitting commits after the fact is not.

I consider this standard practice, at least in the sector I work in (industrial equipment control software, some of which is considered human safety critical).

I'd much rather reduce the risk of mutation to the trunk, by having small easily reviewable commits direct to trunk.

It's less about reviewing commits from a year ago, than making change low-risk today. And small commits can easily be rolled back. The bigger the commit, the more likely rollback will be entangled.

It better to have partial features committed and in production and gated behind a feature flag, than risk living in some long-lived branch.

> I'd much rather reduce the risk of mutation to the trunk, by having small easily reviewable commits direct to trunk.

You're not addressing the problem. You're just wishing that the problem wouldn't happen as frequently as it does.

But that's like wishing that race conditions don't happen by making your allocations at a higher frequency.

I'm describing how Google works with teams with high release cadence, fwiw.

Also, your comment reads a little bit as a non sequitur.

> In 25 years of professional development I’ve never really had a situation where the commits on a branch would have helped me understand what was going on a year ago when the work was done.

My professional experience contrasts with yours. I've even worked at a company where commit history and PRs were so central to understand and explain changes that PRs were even used as the authoritative sources on how to implement features and use frameworks.

Maybe a slight misinterpretation of what I meant. The commit that goes with a PR is definitely useful context, but I’ve found more granular than that is seldom useful. Even big ones like “move from angular to react” - the details of someone getting something wrong in there don’t matter, it’s the scale of it that just makes me go “oh yeah, this is bound to be a mistake”.

Maybe different in other places, but after 15 years in my codebase, I’m still happy with a simple linear history.

> Why in the world would you do squash merges?

Why would you not want to squash merges? It's one of three options offered by GitHub in their PR merge buttons.

They create a linear commit history, which is what you want when you have to audit changes.

> except to clean up messy mini-branches written by total noobs.

Nonsense. You get a messed up commit history as easily as when you create a PR in GitHub, and after team members merge their commits, you click on GitHub's "update branch" button.

You also get a messed up commit history if you merge a PR after someone else merged theirs.

You will always mess up your pristine commit history if you have more than one person posting and merging branches. With one notable exception: squashed commits.

Rebasing fixes all of the problems for which you present squash merges as the only solution.

I want every commit to represent a buildable state in which I have confidence automated tests pass. Bisecting is made unnecessarily difficult otherwise, and it's nice to be able checkout any commit and just build something for testing.

This constraint is usually enforced by code review.

On Github, the unit of code review is the PR.

Therefore, I prefer squashing.

>On Github, the unit of code review is the PR.

It is, and that is some bullshit. The only sensible way to work with that is to break up larger features into several PRs - which is often positive anyway, but sometimes it doesn't fit the nature of the change.

I am not a fan of Github's interface.

But my point is, is that I believe the important thing to preserve in history is whatever your unit of review is. If you could stack PRs and each were subject to the individual review, I would not combine and squash those (just the individual commits within each PR).

>I believe the important thing to preserve in history is whatever your unit of review is

I do not share that belief, but this thread (your posts and others) gave me an idea about code reviews. They became commonplace when I'd been working in software for a while already and, while I see the benefits, I never really felt like I handled them well. I guess one of the ingredients for handling them better is to center the workflow around them - an area in which I have never noticed major changes in an existing project, although they are visible when looking from the right angle: Long-lived, large feature branches have become noticeably less common.

The thing missing with a lot of these branch management posts is release management… because it’s lovely to live in an ideal happy-path world, but what happens when main is tagged for release, only some customers update, main moves of with multiple breaking changes, and only then do some customers require fixes to their releases (who could all be on different i.e even older tags)?

Do you take their tagged release, fix it there, and then send them that branch release with the fix, or do you send them a fix on current main - you know, the main that is now a million releases ahead with multiple breaking changes? And what about all the intermediate release tags? Do you fix each one there too if they have the problem, or do you only update when customers on those releases start having that issue too?

And if you fix to their old tagged release which is incompatible from main, does this mean you have to do this fix twice i.e on their tagged release and also fix it for main? But what if this fix affects other people who are all on different branches too? Now… times this by 20 different customers all running different hardware and different branches with different problems :(

Maybe my comments are off topic, and don’t get me wrong - I prefer “trunk is releasable” motto, but I think maybe as an industry we should all come up with an Acid Test (like the only CSS Acid Tests) so we can through all these branching strategies into the ring

If you need to support multiple versions at the same time, you need to extend TBD in some way.

We just cherry-picked stuff back to release branches, if we needed a fix.

we have a "release" branch and a "develop" branch. The release is trunked on the last released version and (in theory) only gets fixes. If we need to fix a more older version, we create a temporary branch on that version to fix it, and we cherry pick the fixes (or merge to) to release branch and then to develop branch.

The triple mortal loop, comes that we have two versions of the product. One with the old no responsive frontend and other with a modern responsive frontend. And we need to release and develop the two versions for sometime, before the direction decides to kill the old no responsive version. So we end with 4 branches: release, release_rwd, develop and develop_rwd. If we fix something in release, we need to do a diamond merge : release to release_rwd, release to develop, release_rwd to develop_rwd and develop to develop_rwd

Put that in your CLAUDE.MD.

Yeah, I work on avionics software where we sometimes need to support old releases with fixes for many years, and this is basically how we do it. Essentially maintaining multiple trunks and only backporting changes needed by customers using the old releases (there is a high bar so it has to be something like a security fix or an important bug fix)

Trunk-based development fits nicely when you have a single deployment product like a SaaS and you don't need to maintain old versions of your software. You only have one prod environment.

If you build a software that you distribute so people can deploy it themselves (a library, a self-hostable solution, ...), then you most likely semantic versioning. In that case, the best model is to use what semantic release offers.

I've commented about this before.

The answer is development branches are forbidden but releases still use a kind of branching approach.

When you make a release you use commit A from main, then development continues, commit B adds a feature, and maybe commit C fixes a serious bug.

You don't want to make a new release at C because it includes new non tested features, instead you cherry-pick fixes to your release, test the new release candidates, and release that when ready.

Development still happens in main however.

Another big tool to minimise these problems is to separate the concept of feature release from the concept of binary release. You don't have to make a true deployment to release new features or roll them back, just use a toggle switch.

I still have nightmares from my first dev gig. We used SVN, which even in 2015 was prehistoric technology. The server was some awful rustbucket that couldn't sustain 100Mbit. Yes, it couldn't saturate basic 100megabit Ethernet. Everyone had to have big drives in their workstations to maintain multiple local trees. You just couldn't work otherwise.

But probably the worst part was that when I started it was a loose organization where anyone could sneakily merge something into trunk. That changed quickly.

All work was done against a develop branch, and every two weeks, the admin would DELETE THE TRUNK and recreate it from a COPY of develop.

Every two weeks, we lost all history and context for changes in any given release. This was an effort to stop bugs coming back in merge regressions.

Can you guess how well this worked out for them?

That sounds like trunk-based development with extra steps (and only a single release branch that gets wiped every 2 weeks).

Deleting the trunk every two weeks physically hurts to read.

I'm always surprised by how popular SVN still is though. I ported my old Sublime SVN plugin to VS Code years ago for fun, and I still get issues raised today (usually in Chinese, so it seems popular there). If you look at the VS Code marketplace, the top SVN extension has ~1.3m installs.

Bit of a tangent, but it probably wasn't a good idea to release an extension for something I had stopped using long before, particularly when you have to maintain it!

I kinda love SVN, it was a leap up from many popular solutions at the time.

And some people hate it… but, up to and including this story of random-assed deletions, scratch the surface of why and all you hear are self-inflicted harm through bad process.

“Git sucks butt because at OUR shop we start a new repo every few weeks with copy n paste and don’t understand partial checkouts and under-resource the primary host server and also people sneak code into production for funsies.”… … yeah, sounds like “Git” has problems…

stacked PRs. stacked PRs!

Seriously wish the stacked PR workflow would gain more traction outside of FAANG. Apart from the (somewhat pricey) Graphite offering, there's no standard UI for managing stacked PRs in the wild.

GitHub[1] is working through this now, hopefully it leads to more adoption.

[1]: https://x.com/jaredpalmer/status/2019817235163074881?s=20

Damn! Can't wait

I think Gerrit has something like stacked PRs - and incidentally it's used for Chromium and was created for running its own public open source projects by Google. It's also used by the Qt project, which is where I sometimes use it.

Gerrit looks ugly and is not very easy to use at first, but at least it's not slow like GitLab and GitHub.

What's the benefit of stacked PRs?

It's basically automatic rebasing of a set of PRs built on top of each other. Makes it easy to split large changes into manageable pieces that are asymmetrical for your colleagues to review individually, and makes it easy to maintain feature branches long enough to get the feature working end-to-end

The tooling gap exists partly because git's data model has no native concept of "this branch's upstream is another feature branch" — each PR is independent from the forge's perspective, so rebasing one layer in the stack requires manually re-targeting every PR below it. FAANG-internal tools solve this by storing the stack relationship in a metadata layer outside git itself, then regenerating the PR graph after each rebase. Without that layer, the bookkeeping falls on the developer, which is why most teams abandon the workflow after two or three levels deep regardless of how disciplined they are.

We seem to be reaching some sort of consensus among major code review partners on a standard for this information[1]

[1]: https://lore.kernel.org/git/CAESOdVAspxUJKGAA58i0tvks4ZOfoGf...

TBD - its pretty great... aligned also with continuous deployment:

It allows you to get feedback from customers very fast.

It allows you to improve the software very fast.

It allows you to react to the feedback you just got very fast.

Yes, its tricky! You need fast builds, that give you actionable feedback on whether you did a whoopsie.

Yes, it works for all sorts of things: regulated industries, incl finance, embedded systems, apps, websites, ...

Yes, you do need to rethink how changes happen, to look for ways to make that big change into multiple or even many smaller changes, this often has lots of unanticipated benefits.

Yes, it scales to very large deployments and quite large teams.

I worked in a team of four between 2017 and 2020 this way. I really enjoyed it. After that I joined a company that worked with PRs. Felt like such a waste of time.

trunk based is the way to go, especially for small teams building web / backend services

especially combined with monorepo

amount of time people spend updating dependencies between internal services and libraries in a pursuit of semver for now reason is just absurd

Thats my situation. We're 5 devs for 5 years on a monorepo with about 50 services and worked on master without problems. Just from time to time a merge conflict. The only difficulty is to manage Release branch(es) with hotfixes. Discipline, Thinking and Working together, that's the magic source.

Who cares.

Prescribed workflows apply solutions from the wrong direction.

Determine what's impeding your organization, devise a solution.

The last guy I encountered who evangelized something like this had smart frames and a Lovecraft anthology prominently perched in the background of his Zoom shot.

Reply on news.ycombinator.com