Hacker News Reader

Comment by xorcist | Hacker News Reader

Architecture people tend to speak in trivial examples. Swapping out an algorithm with a backwards compatible one? Good for you! It doesn't matter how you do it because it's the most trivial example one could think of. (A more realistic example would be new functionality that touches large parts of the code base, or compliance with a new API that the payment processor switches that has more data on every payment as it crosses our systems.)

I never understood why some people are scared of branching. It seems to work pretty well for Linux development. Is your project really that much more complex? As long as you merge continously from upstream, you're golden. What's the problem you're trying to solve, exactly?

I _do_ have some reservations against #ifdef-sprinkled code. And it's not an unlikely scenario that your feature toggles turn into something very similar, if conditions change and maybe they're not that temporary anymore.

View on HN

> I never understood why some people are scared of branching.

Well this is branching (in the same way that if..else is branching), just not in the VCS. What's the advantage? This methodology can allow code to go to production for some or all users, either as dark launches ( http://agiletesting.blogspot.com/2009/07/dark-launching-and-... ) or as production code. VCS branching gives you none of that, and sometimes you need to roll things out sloooowly, either for quality control or for capacity monitoring.

> It seems to work pretty well for Linux development.

Well for example, if you're following the linux development model for webops, you're almost certainly doing it wrong. I'm sure your website doesn't have release candidates and LTS versions all at different URLs.

> As long as you merge continuously from upstream, you're golden.

This is not really true either. It's fine if you're the only one on a team following that model, but if everyone on your team follows that model, you're delaying collaboration and causing merge debt. Upstream ends up like a ghost town until someone like you dumps a week worth of work on it just before you're about to dump your week's worth too.

> I _do_ have some reservations against #ifdef-sprinkled code.

This is definitely a downside! You have to use a lot of discipline to keep this minimal and well-organized.

I'm not quite sure I understand all of your comment, but I can respond to why one might want feature toggles rather than branches.

One of the activities that can really improve the quality of code on a team is continuous integration. Now, I don't mean setting up a "CI" server to run tests for you when you push your code. I mean actually merging your code with your fellow programmers every 20 minutes or so.

The reason this is such a powerful technique is that it gets people to see what you are doing pretty much as soon as you do it. If you do something stupid (as we all do), they complain instantly because they will merge in your code every 20 minutes. If you do CI well, there are no lingering surprises or hard decisions of "Well, maybe you should start all over again".

The problem is that all the code is mixed up. This is completely fine if you have 1 or 2 week sprints and deploy at the end of the sprint. The problem is that this kills another really power technique called "continuous deployment". Ideally, we would like to deploy as soon as a story is finished. But if it's mixed up with a whole bunch of other code, then you need to make sure that the other code isn't going to break something. Enter feature toggles.

It's a challenge to support both CI and CD at the same time. You can never break any code. It takes a lot of discipline, but if you can manage it, I guarantee pretty enormous improvements in both productivity and code quality (and even programmer happiness). And before you ask, yes I have had teams which did CI well, but no, we didn't do CD.

It's not for everyone. But as I said, it's pretty powerful if you manage it.

You can never break any code.

That's a ludicrous expectation will lead to developers being scared of merging changings, which will in turn lead to them hiding mistakes, refusing to talk through their code, and ultimately hacking things in to the codebase that "work" but are terrible solutions to problems.

It's far better to have a safe environment where a dev can branch a new feature, break the whole codebase completely if necessary, and only merge things back in when they're confident that their solution is a good one (with a review of the pull request by a maintainer or two).

The aim of writing software is to make a good product that solves a problem. It is not to 'never break the build' or 'merge as often as possible'.

> That's a ludicrous expectation will lead to developers being scared of merging changings, which will in turn lead to them hiding mistakes, refusing to talk through their code, and ultimately hacking things in to the codebase that "work" but are terrible solutions to problems.

Maybe, but it can just as easily lead to good practice. E.g. if you're scared to change a particular piece of code, that's a red flag that that particular piece of code is inadequately tested - and "am I scared?" is a much better metric than automated test coverage reports.

But that's the point of a branch, a place to put your code before it's integrated into the whole as an "atomic operation" so:

You aren't possibly breaking live code

You making lots of changes to code other people might be changing, causing conflicts

You don't need to either go back and remove the old branches after the deployment or leave them in to clutter the code.

You don't have to worry as much about overlooking some portion of the codebase using the old code (if you change the API enough to make it incompatible), since you can literally remove old APIs and cause compile errors or easily detectable runtime errors.

You can change data structures with conversion scripts where it's not feasible to overlap functionality (whether those be code constructs, or disk/DB format).

No, that's not the point of the branch.

The point of a branch is to label concurrent development.

There's nothing about a branch that says it must be "test code" or "pre-release" code, but your perspective is common: Everyone wants blue/green or to fully distinguish between "live code" and "test code" because they think the problem isn't theirs to solve: That their manager doesn't give them enough time, or their predecessor didn't like clean code like you do, or that's just what test databases are for.

Infrastructure that really supports branching is uncommon because it's difficult and it requires very careful consideration with regards to the entire architecture. Some examples of systems that actually use branching correctly:

* Implement random 0.5%, 1%, 5% etc traffic diverted to test branches * Front-end web service selects branch from URL or cookie * Backend service compares 5% live queries against test database and compares results

If your system doesn't have something like this, you're really missing out: Really big applications (the kind that span multiple continents) don't have an "atomic operation" to their deploy, and you shouldn't try to pretend that they do. That's how we got IPv6.

Most CI build tools can test branches. They can even merge from upstream on a temporary branch, return the failing tests, tell you when they started failing or being flaky (was it on a commit introduced to master by a recent merge?), then you know for every commit whether it would be green when you merge to master.

I think the calculus changes when you consider teams with 1000s of programmers committing 100s of changes per day. Keeping feature branches up-to-date can pretty quickly adds up to a lot of time and creates a lot of friction for refactoring/cleanup-type changes.

Feature toggles enable some additional cool stuff like partial/incremental rollout and A/B testing, which really pay dividends.

I've never worked on codebases with so many programmers, but I kind of feel that's if that's how you work, 'you're doing it wrong'.

Why would you have so many programmers working on a single codebase, why not split it up in distinct parts? If that's not possible because so much stuff is shared, how can you possibly work effectively with so many people changing so many things?

1000 programmers on a single codebase means:

• "master" always builds

• "master" is always stable

Any testing has to be done with in-code branches, and has to be built in-place incrementally so that other developers can see you touching stuff (and can be defensive about their tasks).

Branching adds too much "meta complexity". That is complexity that you cannot track in actual deployed code. You need processes to manage the branches and to determine when and how to merge. And merging branches is always an error prone process itself. And if you have one deployment target, like a web application, you probably need several staging environments for your branches, so branching does not become the bottleneck towards production. Also if you have test code that is maintained elsewhere (e.g. UI tests), that code cannot reflect your feature changes transparently.

In comparison to that: feature toggles. You need one branch (developers can still can have private branches). And the transitions are transparent in your code. Actually, all the complexity is forwarded to the code and toggle management. Most people understand the concept of toggles better than branching workflows / processes. Toggles are easier to test and your tests do reflect that features are changing.

I made my decision. I will never have official branches again in any of the projects I supervise.

> Architecture people tend to speak in trivial examples.

Even just having a single deployment (such as a web app) is a huge simplification of "deployment" compared to other software such as a driver, firmware, desktop application or just about any other piece of software.

We recently feature toggled exactly your use case - switching to a new API from our payment provider which needs more information.

You seem to be working on the assumption that everything has to be toggled or nothing. What we did was to collect the additional information for everyone (its a low risk change after all), and then slowly roll out the new payment API, starting with staff, then friends & family, and finally doing a phased rollout to customers.

Feature toggling not only allowed us to roll it out, but also to rapidly roll it back to deal with issues, and resulted in a very smooth implementation of a feature that had potential to directly affect revenue.

Has anyone tried Launchdarkly for this?

While it looks pretty slick, I wouldn't personally pay a monthly fee for feature flagging. Our (pretty basic) implementation is just over 50 lines of code, and allows us to do all the targeting we need - either individual users, or particular groups of users we've tagged as having the feature. It just doesn't seem worth shelling out extra for that.

> I never understood why some people are scared of branching.

we are using a similar pattern to add advanced features to certain plan tiers, can't do that with branching on saas deployments - the project isn't complex at all.

we also use it to do progressive release, so that a feature get tested on, say, 10% of the userbase and we improve on their feedback before releasing it at large, but that's secondary to our use case

Actually I would not necessarily oppose toggling and branching. These two method remains tools that can be used separately or together to achieve some goal.

I don't see why one shall not use branching to implement a toggling API.

Well your merging a lot earlier which means less conflicts. Other benefits include, being able to release to a select group of people, and being turn off features easily if something goes wrong

Reply on news.ycombinator.com