Feature Toggles | Hacker News Reader

Feature Toggles

martinfowler.com

66 pointsokfine49 comments

Architecture people tend to speak in trivial examples. Swapping out an algorithm with a backwards compatible one? Good for you! It doesn't matter how you do it because it's the most trivial example one could think of. (A more realistic example would be new functionality that touches large parts of the code base, or compliance with a new API that the payment processor switches that has more data on every payment as it crosses our systems.)

I never understood why some people are scared of branching. It seems to work pretty well for Linux development. Is your project really that much more complex? As long as you merge continously from upstream, you're golden. What's the problem you're trying to solve, exactly?

I _do_ have some reservations against #ifdef-sprinkled code. And it's not an unlikely scenario that your feature toggles turn into something very similar, if conditions change and maybe they're not that temporary anymore.

I think the calculus changes when you consider teams with 1000s of programmers committing 100s of changes per day. Keeping feature branches up-to-date can pretty quickly adds up to a lot of time and creates a lot of friction for refactoring/cleanup-type changes.

Feature toggles enable some additional cool stuff like partial/incremental rollout and A/B testing, which really pay dividends.

I've never worked on codebases with so many programmers, but I kind of feel that's if that's how you work, 'you're doing it wrong'.

Why would you have so many programmers working on a single codebase, why not split it up in distinct parts? If that's not possible because so much stuff is shared, how can you possibly work effectively with so many people changing so many things?

1000 programmers on a single codebase means:

• "master" always builds

• "master" is always stable

Any testing has to be done with in-code branches, and has to be built in-place incrementally so that other developers can see you touching stuff (and can be defensive about their tasks).

> I never understood why some people are scared of branching.

Well this is branching (in the same way that if..else is branching), just not in the VCS. What's the advantage? This methodology can allow code to go to production for some or all users, either as dark launches ( http://agiletesting.blogspot.com/2009/07/dark-launching-and-... ) or as production code. VCS branching gives you none of that, and sometimes you need to roll things out sloooowly, either for quality control or for capacity monitoring.

> It seems to work pretty well for Linux development.

Well for example, if you're following the linux development model for webops, you're almost certainly doing it wrong. I'm sure your website doesn't have release candidates and LTS versions all at different URLs.

> As long as you merge continuously from upstream, you're golden.

This is not really true either. It's fine if you're the only one on a team following that model, but if everyone on your team follows that model, you're delaying collaboration and causing merge debt. Upstream ends up like a ghost town until someone like you dumps a week worth of work on it just before you're about to dump your week's worth too.

> I _do_ have some reservations against #ifdef-sprinkled code.

This is definitely a downside! You have to use a lot of discipline to keep this minimal and well-organized.

I'm not quite sure I understand all of your comment, but I can respond to why one might want feature toggles rather than branches.

One of the activities that can really improve the quality of code on a team is continuous integration. Now, I don't mean setting up a "CI" server to run tests for you when you push your code. I mean actually merging your code with your fellow programmers every 20 minutes or so.

The reason this is such a powerful technique is that it gets people to see what you are doing pretty much as soon as you do it. If you do something stupid (as we all do), they complain instantly because they will merge in your code every 20 minutes. If you do CI well, there are no lingering surprises or hard decisions of "Well, maybe you should start all over again".

The problem is that all the code is mixed up. This is completely fine if you have 1 or 2 week sprints and deploy at the end of the sprint. The problem is that this kills another really power technique called "continuous deployment". Ideally, we would like to deploy as soon as a story is finished. But if it's mixed up with a whole bunch of other code, then you need to make sure that the other code isn't going to break something. Enter feature toggles.

It's a challenge to support both CI and CD at the same time. You can never break any code. It takes a lot of discipline, but if you can manage it, I guarantee pretty enormous improvements in both productivity and code quality (and even programmer happiness). And before you ask, yes I have had teams which did CI well, but no, we didn't do CD.

It's not for everyone. But as I said, it's pretty powerful if you manage it.

But that's the point of a branch, a place to put your code before it's integrated into the whole as an "atomic operation" so:

You aren't possibly breaking live code

You making lots of changes to code other people might be changing, causing conflicts

You don't need to either go back and remove the old branches after the deployment or leave them in to clutter the code.

You don't have to worry as much about overlooking some portion of the codebase using the old code (if you change the API enough to make it incompatible), since you can literally remove old APIs and cause compile errors or easily detectable runtime errors.

You can change data structures with conversion scripts where it's not feasible to overlap functionality (whether those be code constructs, or disk/DB format).

No, that's not the point of the branch.

The point of a branch is to label concurrent development.

There's nothing about a branch that says it must be "test code" or "pre-release" code, but your perspective is common: Everyone wants blue/green or to fully distinguish between "live code" and "test code" because they think the problem isn't theirs to solve: That their manager doesn't give them enough time, or their predecessor didn't like clean code like you do, or that's just what test databases are for.

Infrastructure that really supports branching is uncommon because it's difficult and it requires very careful consideration with regards to the entire architecture. Some examples of systems that actually use branching correctly:

* Implement random 0.5%, 1%, 5% etc traffic diverted to test branches * Front-end web service selects branch from URL or cookie * Backend service compares 5% live queries against test database and compares results

If your system doesn't have something like this, you're really missing out: Really big applications (the kind that span multiple continents) don't have an "atomic operation" to their deploy, and you shouldn't try to pretend that they do. That's how we got IPv6.

You can never break any code.

That's a ludicrous expectation will lead to developers being scared of merging changings, which will in turn lead to them hiding mistakes, refusing to talk through their code, and ultimately hacking things in to the codebase that "work" but are terrible solutions to problems.

It's far better to have a safe environment where a dev can branch a new feature, break the whole codebase completely if necessary, and only merge things back in when they're confident that their solution is a good one (with a review of the pull request by a maintainer or two).

The aim of writing software is to make a good product that solves a problem. It is not to 'never break the build' or 'merge as often as possible'.

> That's a ludicrous expectation will lead to developers being scared of merging changings, which will in turn lead to them hiding mistakes, refusing to talk through their code, and ultimately hacking things in to the codebase that "work" but are terrible solutions to problems.

Maybe, but it can just as easily lead to good practice. E.g. if you're scared to change a particular piece of code, that's a red flag that that particular piece of code is inadequately tested - and "am I scared?" is a much better metric than automated test coverage reports.

Most CI build tools can test branches. They can even merge from upstream on a temporary branch, return the failing tests, tell you when they started failing or being flaky (was it on a commit introduced to master by a recent merge?), then you know for every commit whether it would be green when you merge to master.

Actually I would not necessarily oppose toggling and branching. These two method remains tools that can be used separately or together to achieve some goal.

I don't see why one shall not use branching to implement a toggling API.

> Architecture people tend to speak in trivial examples.

Even just having a single deployment (such as a web app) is a huge simplification of "deployment" compared to other software such as a driver, firmware, desktop application or just about any other piece of software.

We recently feature toggled exactly your use case - switching to a new API from our payment provider which needs more information.

You seem to be working on the assumption that everything has to be toggled or nothing. What we did was to collect the additional information for everyone (its a low risk change after all), and then slowly roll out the new payment API, starting with staff, then friends & family, and finally doing a phased rollout to customers.

Feature toggling not only allowed us to roll it out, but also to rapidly roll it back to deal with issues, and resulted in a very smooth implementation of a feature that had potential to directly affect revenue.

Has anyone tried Launchdarkly for this?

While it looks pretty slick, I wouldn't personally pay a monthly fee for feature flagging. Our (pretty basic) implementation is just over 50 lines of code, and allows us to do all the targeting we need - either individual users, or particular groups of users we've tagged as having the feature. It just doesn't seem worth shelling out extra for that.

> I never understood why some people are scared of branching.

we are using a similar pattern to add advanced features to certain plan tiers, can't do that with branching on saas deployments - the project isn't complex at all.

we also use it to do progressive release, so that a feature get tested on, say, 10% of the userbase and we improve on their feedback before releasing it at large, but that's secondary to our use case

Branching adds too much "meta complexity". That is complexity that you cannot track in actual deployed code. You need processes to manage the branches and to determine when and how to merge. And merging branches is always an error prone process itself. And if you have one deployment target, like a web application, you probably need several staging environments for your branches, so branching does not become the bottleneck towards production. Also if you have test code that is maintained elsewhere (e.g. UI tests), that code cannot reflect your feature changes transparently.

In comparison to that: feature toggles. You need one branch (developers can still can have private branches). And the transitions are transparent in your code. Actually, all the complexity is forwarded to the code and toggle management. Most people understand the concept of toggles better than branching workflows / processes. Toggles are easier to test and your tests do reflect that features are changing.

I made my decision. I will never have official branches again in any of the projects I supervise.

Well your merging a lot earlier which means less conflicts. Other benefits include, being able to release to a select group of people, and being turn off features easily if something goes wrong

I find the `scientist` gem by GitHub [1] a very nice way to approach the problem of testing alternative implementations.

Basically `scientist` always runs both the original code and the new code.

  def allows?(user)
    science "widget-permissions" do |e|
      e.use { model.check_user(user).valid? } # old way
      e.try { user.can?(:read, model) } # new way
    end # returns the control value
  end

It returns the result given by the original code, but it also compares both results and store statistics about when the two codes produce different results. This allows you to run experiments directly in the deployed branch without any service disruption.

[1] https://github.com/github/scientist

A more powerful version of these things are called dynamic variables, and they're worth some study beyond feature selection and testing: Otherwise you just get global variables with funny names.

Many languages have some dynamic variables (this, self), and few languages have first-class support for them (notably CL, Perl). While dynamic variables are tricky to emulate fully, as long as you have the ability to catch exceptions and closures, a very fair simulation can be made:

    (function(){
      var o={};
      D=function(k){return o[k]}
      dlet=function(b,f){
        var r,s={};function oops(){for(var k in s)o[k]=s[k]};
        for(var k in b)s[k]=o[k],o[k]=b[k];
        try{r=f()}catch(e){oops();throw e;}; oops();return r;
      };
    })();

Besides feature variables that are globally accessible, one useful thing you can do with dynamic variables is set up error handlers. Consider the situation where you are saving a big file to the disk and run out of space. If you:

    save_stuff(); if(oops)throw 'out of space'; save_more_stuff();

then your slow saving operation needs to be retried from the start, however if you use a dynamic variable to look up the handler, you can:

    save_stuff(); if(oops)D('out of space')(next); else next();
    function next(){ save_more_stuff(); }

The user can then use their fancy multitasking environment to clean up some space, and we can continue our operation. I generally recommend this form of error handling anyway.

Another useful thing is for context: Imagine you have a user interface choice between an HTTP response and a command line interface. One way to do this is to have two applications, and another way is DI (dependancy injection: pass the "response" object around), however another way is using dynamic variables:

    D("output")(...);

This has the benefit of not requiring an extra argument to all of your functions.

This happens more than you might think: Many people when confronted with all their dynamic variables put them into some kind of "master object" (current logged in user, output handler, database settings, etc), however dynamic variables

When you do this, a lot of features become very easy to set:

• Capturing output (simply mock a new "output" var)

• Testing logic: dlet({db:test_settings},r)===dlet({db:live_settings},r)

• Impersonate users (if the right credentials are available)

And so on.

At IMVU we had feature toggles rolled into the A/B Experiment system. It proved to be a vital part of Continuous Deployment.

Basically: Wrap every new feature in an Experiment Toggle. Deploy to production (once ready for 'non sandbox' testing). Turn on for test users, test. If good, start the rollout to experimental users. If after weeks the numbers worked out, turn it on for everyone...

...but if at any point unexpectedly bad things happened, just flip the feature toggle switch. Most of the time if "bad things" were happening in a relatively new feature, this would stop the panic and let analysis of the root cause of the problem happen without the pressure of everything being on fire.

Same at Facebook https://www.quora.com/How-does-Facebooks-Gatekeeper-service-... with rollout UI similar to what advertisers see on ad purchasing UI (by country, college, demographics, etc.)

One point to be aware of with feature toggles: if you're using them as a substitute for branching and merging in your VCS, you're deploying code into production that you know for a fact to be buggy, insufficiently tested, and incorrect. If your feature toggles themselves have bugs, or don't properly isolate your new code, you could easily run into problems up to and including data corruption. You may be switching your controllers on and off, but are you also toggling your static assets such as HTML, CSS and JavaScript?

People are often scared of feature branches because of large, tricky merge conflicts. However, large, tricky merge conflicts warn you (noisily) of problems such as these before your code gets into production. Problems with feature toggles, on the other hand, don't show up until after your code gets into production, by which time it may well be too late.

Overall feature toggles are an interesting idea. I'm not quite as adverse to long lived branches as some seem to be, especially with CI systems that can automatically detect merge conflicts when they first occur and run automated tests on the merged code base.

However, this part seems strange to me:

> The team particularly appreciate that this will allow them to test their new algorithm without needing a separate testing environment.

Even with feature toggles, I would never want to test a new feature for the first time in my production environment. Sure, the feature isn't on for everyone. But it still can have a performance defect that brings the overall system to its knees or a functionality defect that corrupts the data store for everyone.

I find it amusing when CI tooling is used like this, to allow people to keep changes isolated in separate branches with less pain from delaying the integration.

It's more Continuous Isolation than Continuous Integration.

There are ways to test features for performance/accuracy in the production environment with reduced risk. One approach I've used successfully is branch by abstraction with verification.

http://www.alwaysagileconsulting.com/articles/application-pa...

Essentially you first extract a common interface for the component you are replacing. Then you release both versions and send (all or a percentage of) input events to both the old and the new implementation. You discard the response from the new implementation and continue using the old codepath for responding to users / performing calculations, but importantly - you compare the old and new results and alert/log if they differ.

This allows you to gain confidence in a new implementation's behaviour in the production environment and integrate your code with the rest of the system with significantly reduced risk. When you're happy you can flip over to the new implementation and delete the old.

There are also various other techniques for reducing the risk of datastore corruption that you mention. I've written about some of them here http://benjiweber.co.uk/blog/2015/03/21/minimising-the-risk-...

Most of these risks are actually smaller with more frequent and smaller releases of functionality into production. Releasing more regularly also encourages you to think about how to properly mitigate these risks.

There is a substantial section of the article about modifying their high level tests to test both branches and keeping the toggle off in production whilst turning it on for some exploratory testing.

> I'm not quite as adverse to long lived branches as some seem to be, especially with CI systems that can automatically detect merge conflicts when they first occur and run automated tests on the merged code base.

This system of branching is clearly the best way to maximize large merge conflicts, if you think about it.

Let me restate this: The main reason that you branch (to avoid conflicts with other devs) is the single scenario that this system does the absolute worst at.

Conversely, if you're not having problems with this system, you never needed it in the first place.

Oi vey.

Coming from the good old C I am frankly baffled how some triviality like this one can be wrapped into so many fancy words and a full article. This is like hearing that taking your shoes off when entering home is a good idea. Do you know why? Let's start with an example. Picture this - it's raining outside, water soaks the earth and some of it may get on the soles of your shoes... you get the idea. It's a subject hardly worth a passing mention if any. "Feature toggles". Bah. Makes you wonder what will happen when they discover function pointers or, god forbid, the config files. Now that will likely be worthy of at least a book.

An interesting way to implement feature toggles is with your existing A/B test system. Basically, create an "experiment" for each new feature. Set the initial "treatment" size to something like 10%. Once the feature has been validated in production, set the treatment size to 100%.

It's scary that you have to explain this. This is A/B testing 102, all new features are experiments.

A few months ago there was a great blog post by the Instagram team on their particularly advanced flavor of this for their continuous integration. Highly recommend: https://engineering.instagram.com/posts/496049610561948/flex...

Whoa. That's an ugly URL! b^) I clicked through just to see if I could cut out the random number, but no dice.

It seems the 'random' number is actually the important bit. You can remove the `/flexible-feature-control-at-instagram/` part, which is only there for SEO (presumably). StackOverflow URLs work the same way, in fact...

I really like the idea of feature toggles but i'm scared of quickly having a lot of technical debt because of them.

I was looking at using something like https://launchdarkly.com but the pricing is very expensive even at fairly small scale.

Co-founder of LaunchDarkly here-- we're dropping our startup package to $9 / month in the next week. Come check us out.

If you have a RBAC setup it's pretty easy to setup feature testing. It's the second most awesome reason why I setup RBACs on any and all projects.

We use feature toggled a fair amount on our SAAS application. It does create technical debt which at some point has to be addressed.

One thing that confuses our junior developers is the difference between a feature toggles and a client permission. Often they misuse a feature toggle as a client permission, simply because only one client is using that feature (currently).

How do you typically address or mitigate your technical debt?

Currently we are forced to ignore it until it becomes a performance problem or a new feature touches the same code. This is simply because new features are paid for by clients and they take priority over technical debt.

We only tend to refactor code (as a rule) when you are already working inside that code section on something new (feature, improvement or bug).

Otherwise junior developers have a tendency to want to refactor all the things!

If you wrote a new unit test, or altered an existing one to accommodate new code, then knock yourself out and refactor away and knock yourself out!

This all thing reminds me why I love so much so called Enterprise Developers and Martin Fowler with Thoughtworks and their articles (no offense, thanks to Martin for the idea of refactoring, but such things are wrong). Seriously, they take simple idea reducible to more general principles which is not a very big deal and start build ad-hoc ideology and terminology around it: Hm-m, some abstraction around feature flags config? I think we should call it "toggle router". Then people start to create frameworks (not libraries) for this and after all technical recruiters will end-up searching for people with particular "toggle router experience".

Feature toggles are useful but should be used with great care, especially in complex codebases. There are challenges to using them. If you aren't careful or don't have very good test coverage you can break unknown sections of code.

The other pain is removing them, again if not done carefully or with very good test coverage you can unknowingly break things.

I am pretty disapointed by the limited use cases pictured in the article. I pinged the author without success.

Check at bottom of ff4.org page 10 others.

E.g : Deliver your product with toggles off on each node of your cluster and when all are ready : toggle on - no more inconsistency during release - no rollback as you simply toggle off if not working.....

Ah, isn't this just... an option?

There was a nice talk about feature toggles at RubyConf 2015: https://www.youtube.com/watch?v=rBBLMmr9e-k

Configo.io can do that for your mobile app

Configo.io can do that for your mobile app!

It's interesting to see this here now, because ~10 hours ago, when it came up in my feedly stream, I tried to submit it and only got the message that it's already submitted. I searched here for it but could not find it, though.

Thanks for the downvotes for being curious how Hacker News works. :(

Reply on news.ycombinator.com