Branchless development | Hacker News Reader

Branchless development

tedunangst.com

77 pointsluu59 comments

Branching or not is somewhat of a red herring. If you are working on a commit then while you haven't pushed that commit, you're effectively working on a branch. If somebody else works on the same code at the same time, then those changes need to be merged. Whether your VCS does or doesn't call that a branch is immaterial. What matters is that two developers can work on the same code at the same time, and that you need to solve concurrent editing conflicts (logical ones, not just the textual ones that your VCS will find).

The pertinent question is how often you merge. On one end of the scale you merge on every keystroke. Every time Joe types a character it gets put into Anna's files. Clearly this doesn't work. On the other end of the scale every developer works on their own copy, and all the work gets merged just before a release. Clearly this doesn't work either. There is a scale from maximal disruption minimal effort merging to minimal disruption maximal effort merging. You want to find a point on that scale that minimizes the total amount of work.

This is super-important; it is far better for a developer to push a non-working feature on a feature branch than for them to leave it on their machine. They ought to do whatever the equivalent is of a git rebase every morning or evening, so that it is still at least nightly integrated for them.

Keep in mind though that if everyone is working on something that takes multiple days to complete (and your definition of non-working is incomplete) then all the work is happening in feature branches and master is quiet. Those rebases are easy because no work is going on there.

Feature toggles remove this problem by keeping everyone in trunk/master. You can then have CI builds with whatever set of toggles you want.

Feature toggles are a good option too; I have used that before and like it. The downside is that you can end up with a giant set of feature toggles and very little idea of which combinations work.

Use automated integration tests and test every combination.

Exponential runtime.

Don't overdo it, use coarse grained switches.

Throw hardware at it.

Switches are temporary, with time their number will settle.

The one place I worked that used them, switches were not temporary. There were about 4000 of them last time I checked.

There was an advantage: if a bug popped up you possibly had a workaround immediately for the customer. Downside is you had no idea if that workaround would expose more bugs.

"It's just temporary" is one of the five famous last words (together with "it worked on my machine"). It can easily turn into one of those projects where someone just shoved in autoconf to turn them into a giant ball of IFDEFs so nobody really knows which combinations work.

Toggling also only makes sense for the cases where you don't change the structure of your data. So be careful, and make sure you have a plan if you decide to do it.

It sounds like you need some way of applying downward pressure to the number of feature toggles, so they get cleaned up when no longer relevant. I don't have an immediate suggestion, though...

..and the circle is complete.

If you're a fan of this workflow then I can recommend using Visual Source Safe. It even has exclusive locks which completely eliminate merge conflicts.

I've been building software professionally for almost 20 years, I've seen (and experimented) with almost every VCS/SCM strategy imaginable.

My conclusions:

This only really works for tiny teams - 2-3 people who work very well together. It needs high bandwidth communication.

For teams of 3-10 short lived feature branches are effective with a little bit of planning to minimize merge conflicts.

For larger teams you really have to split the system into self-contained modules or microservices, each module having clear owner. These modules are then shared as source, library or service depending on the actual needs.

I've never had good experiences with longly lived branches. They can work whilst adding experimental modules which require only a couple of tiny API changes to the main system; but then splitting the module into a separate project/repository and making the API changes (behind an #IFDEF) makes more sense.

same here. the only long lived branch we have is the deployed code branch, which I found more convenient than just a tag

it also matter average team skill. if you have to make do with what you find/can afford, then feature branches are good at isolating devs from each other non working commits.

Sorry, but your experience of Visual Source Safe is totally different than mine. I started working at a small company in 2002, where they used Visual Source Safe. Next to the VSS database file getting corrupted every few weeks, the 3 developers were always asking each other to unlock a certain file because someone needed to make a change in the same source file (but completely different method/location).

I introduced Subversion. No more corrupt files (shiver), it just kept on running. And you could actually work on any file. If you divide your work properly, you can work in the same file without working in the same code. And if the small chance of changing the same line occurs, the merge wasn't that complex to fix.

We never looked back, and therefore I am completely disgusted that someone actually suggests using such an unstable, non-efficient tool.

I suspect there might have been a touch of sarcasm in that VSS remark

Yes, I was being sarcastic. Sorry for confusing you koonsolo.

Visual Source Safe is an absolute pig, unless you're already using it and can't switch then I strongly advise getting as far away as possible.

Team Foundation Server - the replacement - has historically been just as poor. It does however have the advantage of including some extremely powerful software lifecycle management features.

Recent developments - such as the Git-based Team Foundation Online - are really rather good, especially if you need more advanced features and like giving Microsoft money ;)

I agree with the VSS-hate, of course, but it's unfair to say that TFS is just as bad. TFS is annoyingly difficult to administer, but it's a zillion times more robust than VSS, and compares favorably to Subversion or Perforce (and has excellent Visual Studio integration).

Git totally kills it, of course, but as traditional old-school VCS goes, it's not bad at all.

> It even has exclusive locks which completely eliminate merge conflicts.

I assume those locks can be overwritten by an admin, right? I'm exactly the sort of forgetful jerk who'll lock an important config file before going on a two-week vacation.

In my experience, if you have high bandwidth communication (e.g. a team room) then trunk based development scales to well beyond 2-3 people. I've seen it work on at least one large distributed teams (3 team rooms, 3 time zones, ~30 developers)..and it regularly works on teams of 6-10 developers.

Regarding VSS and exclusive locks. No. Trunk based development with no exclusive locks is fine. Pessimistic locking was unwarranted pessimism when applied to source control.

Also, note that Google is trunk based...with a single trunk across the whole company. I think Facebook is too?

In that case you were absolutely having merge conflicts - only you were forced to resolve them the second you updated your local working copy.

Over the years I've lost so much time to small errors resolving merges - particularly the type which involved cross-file refactoring (where two people have moved methods and then refactored them).

The beauty of using branches - especially with Git - is that you as developer gain a lot of power over how you apply merges. The "pre" state is also under version control, so you can always go back. Plus if you're using Git then you're implicitly doing branch-based development (git clone creates a local branch).

My current team of 6 developers switched from TFS trunk-based to Git with short-lived feature branches and they've not looked back. We've greatly reduced merge issues and downtime caused by bad check-ins. Both still happen, only less than before.

Of course we still have merge conflicts. You do too. Or is your work just organized such that no two people are ever working on the same file concurrently? If it is then you'd not have any merge conflicts with trunk based development either.

Fundamentally merge conflicts are unavoidable.

The claim by the trunk based development crowd is that smaller, but more frequent merge conflicts produce fewer mistakes. Trunk based development leads to smaller merges because you do them more frequently. That's about it.

Working on the same file concurrently does not guarantee conflicts - working on the same code does.

We recently switched to short-lived feature branches (short-lived as in a couple of days maximum), and I haven't had to resolve merge conflicts in months. Our feature branches previously used to last around two weeks and merging things in was collectively not looked forward to. Now I don't even have to give it a second thought.

tl;dr not having merge conflicts is an entirely possible happy reality.

Sure. My point is that avoiding merge conflicts has nothing to do with your branching strategy...it has to do with what work you parallelize.

> Also, note that Google is trunk based...with a single trunk across the whole company. I think Facebook is too?

Yep!

http://paulhammant.com/2013/04/05/what-is-trunk-based-develo...

And still somehow feature switching works great at Facebook, Google and Goldman Sachs with big teams.

Merging complex changesets across branches is impossible to do 100% correct every time. Not unless you're a 10 feat tall code crunching robot.

So if I'm understanding correctly, this terribly written article (he tries to make a point about merging but doesn't actually make it amongst other points he fails to even state) suggests using feature branches inside the code instead of feature branches within source control. It also requires mini code reviews and merging at every single commit. You cannot get rid of merging. Instead of doing it once when merging a branch, however, you're now doing it every commit. Since you're merging smaller chunks, it's impossible to see the bigger picture of the whole feature as you merge/review, and it removes the ability to do meaningful code reviews on features before they actually get into the main code base. Not only that, sometimes you end up in a race with your teammates, knowing that you're working on the same piece of code and that whomever checks it in last will have to merge. Stupid, but it can happen.

To me, the ability to have logical separation of the features being built and do meaningful code reviews far outweighs any proposed speed benefits of not branching, benefits that I assume show up only in larger groups as they certainly don't show up in small groups. Which bring me to my last point, the "slow is fast" section, which doesn't provide any evidence that slow is fast. Other than users not having to decide which branch to run (which takes all of two seconds, BTW), development is much faster with feature branches and the other negatives do not show up in my experience. What I'm trying to say is that "fast is fast." Slow is not fast and the author doesn't make his point there either.

tl;dr: All in all, he points to no actual pros to branch-less development.

Heh, there's always opposition to new practices. Branching has been mainstream for a long time, it's only fair to expect resistance to any alternatives.

Give it a few years and everybody will use feature switching, except the old man enterprise dev :)

There are a million things that feature switching let's you do. Enable and disable instantly, enable for .00001% of the users, integration test different combinations, etc.

https://secure.phabricator.com/book/phabflavor/article/recom...

In my experience, branching has only been mainstream since git became popular a couple of years ago. Before that, when using svn and other similar systems, branching per feature was unlikely in my experience and a major pain.

Branching is very well supported in Subversion. There is a reason feature branches occupy a large part of the SVN Book.

I would say feature branches were mainstream usage of CVS too, but SVN made it more popular because merging is more effortless.

Github probably popularized it, because they put a web interface on it (which is great), but that can be said for a lot of things.

That's fine until you have a 4 MLoC behemoth with 50 developers (I use that term loosely) that isn't separated into logical concerns properly like your average "enterprise product".

I've seen those things and what you end up with is a dev team in tears because some chump has fluffed the source tree, the integration build is still running so no one else knows and checks out the trunk followed by boom and 2 days of paralysis while everyone works all their changes back into it for a checkpoint build.

All mature products tend to evolve into this state unfortunately.

Ergo, that works on small things but doesn't scale. Minimally, using feature branches and integrating them in turn is the only situation that works on those sorts of products.

Facebook seems to be managing with 1k+ developers and 50+ MLoc (https://secure.phabricator.com/book/phabflavor/article/recom...).

I'm not entirely convinced either, but the argument isn't without success stories.

People in academia have proposed a number of tools & approaches on how to enable concurrent development while mitigating the "risks" of branching.

To name just a few:

* Palantir by A. Sarma et al. [1]

* Crystal by Y. Brun [2]

* CloudStudio (project I worked on); Video demo: https://youtu.be/R3Fz0Tcdz0Y

To the best of my knowledge, however, none of these projects have found any widespread adaption. A lot of it also has to do - I think - with culture and personal preference.

[1] https://scholar.google.de/citations?user=shMjCasAAAAJ

[2] http://homes.cs.washington.edu/~mernst/pubs/vc-conflicts-fse...

People thinking and saying that this only works for small teams should take a look at how many devs are listed in OpenBSD's latest release announcement.

Hint: it's more than 2, more than 3, and more than 50.

How many have direct commit access? If you're sending patches over email, guess what: you're doing merges, whether you call them so or not.

Or, say, Google and Facebook?

http://paulhammant.com/2014/01/08/googles-vs-facebooks-trunk...

I'm also a big fan - for a tight team anyway, might be harder for a more distributed project - you should eat your own dog food - more incentive to not check in crap and an incentive to fix broken stuff fast.

Having an autobuild system to build every checkin is a big plus.

I worked on a chip once where we decided to make everything a branch, at one horrible point we didn't have a mainline, we didn't know what we were building, we had to put people on full time merging trying to figure out what to test.

The thesis of this post is that merge conflicts basically negate all of the other benefits a branched workflow provides. I'm not sure how the workflow the author suggests deals with concurrent changes to code otherwise though, and I feel like merge conflicts are preferable to changes that overwrite other changes. Not taking advantage of branching destroys much of the benefit a DVCS provides.

In order for this workflow to be reasonable for any development team of a size greater than about 3, each commit would need to move all the code in to a working state. While a reasonable pattern in theory (Don't TRY commit breaking changes!), it is frequently useful to track code in an intermediate state. Sometimes (read frequently), this code may not be representative of the final state of the technology or may not even work as it should. Not maintaining individual branches means that these changes are exposed to the entire team and can be very very messy.

All in all, what a silly approach to revision control. Take advantage of the latest patterns and a system that provides the loosest coupling and highest distribution. If we do as the author advocates, we might as well all just switch back to CVS...

We have a team of 50 and single branch works well for us.

> merge conflicts are preferable to changes that overwrite other changes

Nobody is saying that. The difference is really merging as you go vs doing one big merge at the end. I personally fall on the side that it's far easier to merge as you go.

You merge as you go in branched development too?

Master is typically periodically pulled into the feature branch to ensure you're working against the most current version of the codebase.

EDIT: Yes, this can create an ugly revision history, but that's what rebasing is for, right?

We rebase main branch into feature branches constantly. We rebase to squash reviewed features, and then ff down to the main branch to integrate the feature. If something doesn't integrate well, we commit the appropriate fixes locally, squash locally (if necessary) and pull/rebase/repeat on changes.

It sounds crazy now that I write it down, but it really keeps our commits clean.

If everyone is working in feature branches then you're still merging less often. Until you've merged your work into master then I'm not merging it into my feature branch. That leads to bigger merges than if everyone is committing to master all the time.

Pros or cons of each aside, advocates of feature branch based development should recognize that they are merging less often than they would if they were doing trunk based development.

Branching/merging in DVCS is only easy if there are no merge conflicts.

Checking in code to the VCS that makes the tree NOT build will get you screamed at (at least within the OpenBSD project). This requires proper planning for big changes, and to break them down into small, easy to review, patches.

OpenBSD uses CVS as main repo because it fits their development style. Bevore you knock down CVS look at AnonCVS and cvsync. http://www.openbsd.org/papers/asiabsdcon2009-release_enginee...

I'll fully accept that it's Teds experience that this works for OpenBSD. I'd be hesitant to draw any conclusions from that concerning the viability of the same process for any other project or organisation.

An open source project with highly motivated and capable developers and a benevolent dictator in charge functions differently from other projects and organisations. A process needs to be attuned to the people, and the relations between those involved, involved.

For long lived branches, I prefer to track and review the work done on master and merge master into the branch every often, I do every day. You'll handle the conflicts more easily, as they come and you'll get fresh insight/explanation from your coworkers. All at once at the end, and you end up in a nightmare of "Do you remember when you XXXX N weeks ago ?"

Diverge and merge at the end is quite utopist on complex projects.

It would be nice to have some kind of monitoring system that warns me when another user is editing (or has edited) the same file(s) as me. Then we could talk, or one of us could wait, to prevent merge conflicts in the first place.

Of course, this would be even more useful if it only alarms the users whenever they have created a merge conflict.

Branchless development is a terrible idea. I think what the author is wanting to avoid is long lived branches. Keep your branches small, isolated, get your changes into the main branch as soon as possible, but develop each atomic change independently of it. We do this at Gumroad, keeping our branches as small as possible, and it's amazing.

I work without branches but with a large number of good tests, and it works like a charm. On the other hand, I know people who can spend several days to merge their one week changes back to the master which is a great waste of time and really boring.

We make tests to solve subtle merge bugs like the article's example about dialogs, and watching what the other team members are merging help miles too.

How would this development style be compatible with code reviews?

It's not, in my view, unless you count pair programming as a sufficient substitute.

I've seen branch-and-review used in anger. It can easily go bad: it drastically multiplies the inventory of work-in-progress. Everyone becomes blocked on someone else, moves on to another task, then loses context on the previous task or tasks. Stories with soft dependencies wind up sprinkled across several branches, making user testing a bit of a merge-and-pray event.

It sounds great. Reviews! But efficient!

And it turns out to be a tangle of interlocked gears.

I imagine somebody has made it work. I have not seen such a case.

My personal preference is a single development branch^, test-driven, pair programming, with rebase and integration test before pushing.

[^] If it's a private project, this is master. On github, the master branch can't be used as a true master, because it is a de facto release branch.

Isn't that a little at odds with the idea that the author is describing some variant of the OpenBSD development process (unless I misread the thing) and beside being a large and ongoing project, they've been reviewing code since before it became fashionable?

The original literature on reviews that gets everyone hot and bothered describes something very different from what is done in practice.

Formal meetings with scribes and reviewers and procedures and checklists are not the same as "here, check out this patch and merge it if you like it".

I haven't kept up with research and I don't know if anyone has shown that TDD + pair programming is a reasonable substitute for full-blown code reviews in terms of bug yield.

Now that I think of it, what I saw was an inversion of the classic CVS commit-bit model. CVS-based opensource projects typically have a praetorian guard who review everything before it gets committed.

What I've generally seen is the reverse: soliciting opinions from other teams with little context or familiarity with the original codebase. So they take days or weeks to respond and when they do, it's quite cursory.

Sounds to me that we work for the same company :)

I bought into feature branches and all that stuff right after uni and tried to apply it at my first job without all the success I was expecting. Then when I moved to my current job in which we do TDD, Pair programming and single branch development I was a bit reluctant. After almost 2 years I haven't come up with a single scenario in which this approach has given us any troubles and in hindsight I think is what I should have proposed at my first job.

Until I got to Pivotal Labs, developing on a single branch seemed impossible. Now it seems obvious.

That said, I'm anchoring a project where my predecessors chose a feature-branch model. I have a motto that "sometimes it's better to be consistent than correct". Done properly, it's not too hard, but in practice that's because we don't have a large team and we still rebase on and merge back to master frequently.

That's a funny saying. I have a very different saying. It goes like this: "It's always better to be correct than consistent."

I will give you another one:

When in Rome do as the Romans.

I think consistency is a best practice with precedence over all the other ones. Having a team agreeing on something and being happy with it is sometimes far more important than that something being the best of the possible choices.

Let me expand.

The perfect is the enemy of the good.

It's unethical for me to put my own standard of perfection ahead of delivering business value and soundly engineered software.

My peers have opinions, formed validly, from their experiences. I have my own. Sometimes they differ. Sometimes I just need to suck that up and get on with the job.

Big changes (mostly) need to be split into small and easy to review patches. There are almost no commits without at least one OK from another main developer.

Developer changes code and create a diff. This diff is sent to the mailing list or other developers to test and review. The patch only gets committed when other developers have ok'd it.

Why would it be a problem? Actually it is favourable, since the reviewers always see the actual changes to the code base in each commit, without having to deal with merge commits.

Reply on news.ycombinator.com