When Feature Flags Do and Don't Make Sense (2019)
software.rajivprab.com
software.rajivprab.com
I also seen condensed flags where the flag record embedded which IDs were activated, with a massive overwrite risk anytime someone touched any setting.
Also flag-averse modules where there was so much magic that one one could ever understand exactly what would happen until they loaded up a real or similarly structured set of records. And they didn't trust their results for more than a month or so because changes were frequent as things needed to evolve. The case for this craziness is that too many flags means folks will overlook things, or we'll forget to make the new shiny on-by-default after the roll out period.
IME there is a balance between every feature and code path getting a flag and nothing ever does. Of course the flags themselves introduce complexity and risk. And there's the work to remove them with the vestiges and QA that change for regressions.
To this day I've never found any good guidelines for flagging changes like this without resorting to widespread file duplication, abuse of OOP, or an ad-hoc versioning system. It comes as no surprise that nobody in my organization was willing to do this sort of work.
I think the true cost is that feature flags can cause code bloat and readability issues, since engineers typically aren't great about cleaning up feature flags after things have been rolled out. I think that's an easily solvable problem that doesn't really necessitate a scarcity mindset of "just use less feature flags / only when necessary" though. LaunchDarkly makes it pretty easy to track feature flag usage and remind people to clean up old ones.
Not quite. When you add a feature flag, you now have a system that has N separate code flows that you need to verify before launch,and then have to modify and retest to remove the feature flag. A feature flag ends up doubling the workload to test and verify the feature.
> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
This is perhaps the most relevant point that the blogger misses. Feature flags are runtime switches that can extend simultaneously to multiple systems and even clients. You can't pull features from clients with a release, where week-long release processss can barely get you a 60% uptake. Redeploying away features in the backend can easily take half an hour, specially if it's a system that does multi region rolling deployments.
If you want instant switches, your best option is switches flipped at runtime. That's what a feature flag is.
I would argue that if flipping a feature flag isn't faster than doing a rollback, your feature flags are not functional. They should be essentially instantaneous and the first thing to reach for as soon as you've identified a problem.
> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
This I totally agree with. Using feature flags well requires some discipline on the team's part to stay on top of them and remove them once they've served their purpose.
Feature flags are terrible. Working on a crash / memory corruption in an optional subsystem? You disabled it previously for your local build, and you'll lose hours failing to repro despite QA giving excellent repro steps.
(For my own gamedev background, I learned to mute audio by setting volume to 0 instead of by disabling the audio subsystem.)
There is some confusion in this post. For risk management, feature flags are used to mitigate the risk of issues not being caught by the QA process. Wishing for a QA process to be 100% flawless doesn't make it so.
Also, guess what an incremental deployment is.
I think the blogger tried to make sweeping statements but didn't actually spent any time thinking through the arguments.
The reason why rollbacks isn't sufficient for us is that our service is semi-stateful (postgres connections are stateful, we proxy those connections). Because of this, we always keep around old pods for 5 days to let connections drain.
A deploy+rollback ends up with 3x the pods lying around, and if we deploy a fix patch that's now 4x - and if we don't deploy the fix we have 2 weeks of changes piled up for the next release.
Because of this, we instead use the feature flag. We can toggle it on and off very quickly for risky changes, and it makes no changes to pod counts
This design seems rather brittle. What happens if a node spontaneously fails, or you need to move to another AZ? (Assuming you’re in the cloud.)
Node failure is indeed a problem. Not much we can do there. Fortunately it's rare enough historically.
We are working to split the code so that the process that holds tcp connections isn't the process that handles the logic though, so it's easier to update inplace
Well this all makes sense now. My enterprise laid off most of our QA team and I see a constant stream of rollback announcements. Who needs regression testing anymore?
And that brings us to another great point, which is too many flags is problematic. So often there's an excuse made in the nature of, "we'll go back and remove this later," but later never comes.
At the end of the day, it's rigor that separates good teams from bad teams. Good teams will rigorously review old code and remove it; it's all too easy to do the opposite.
I doubt anyone making this sort of claim has any professional experience maintaining any sort of user facing software.
Features that require cross-system support can't easily be pulled out with a revert, particularly in CICD systems where cherry picking a revert can easily be incomplete/miss a bug fix and reverting the whole history will end up inadvertently pulling out some other feature.
Also, a deployment takes time, particularly in services with more than one deployment region. Moreso if n>1 separate systems are involved.
In comparison, runtime config changes are enforced in a fraction of the time, and pulling out the code just involves a roll forward. Simple.
I'm sorry, but you're completely wrong. I know for a fact that Amazon uses extensively a A/B testing system for feature flags. Others in this discussion already mentioned it.
Source: worked at AWS for 7 years.
And knock it off with the attitude.
1 feature flag = 2 behaviors 2 feature flags = 4 behaviors
So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
This is specious reasoning. When you add a feature flag, your goal is to purposely introduce version variations. How come is your explicit goal framed as this sneaky gotcha?
And let's be serious for a second. Do you see this "hockey stick growth" as an issue when you look at user settings?
> So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
Not really. Feature flags have a life cycle which is managed by release managers, and the goal is to always get rid of them asap. They are introduced to manage the risk of a rollout, and when the transition is over you pull the feature flags. Done.
Everyone intends to collect the garbage. Few actually do.
I’d rather invest in the feature flag upfront so I can unbreak customers in 2min rather than telling them to wait hours for the rollback.
Why are rollback and release so slow? Because we’re deploying to 40+ availability zones world-wide, totaling many thousands of servers, and we don’t want to worsen the impact by doing an accelerated “everything-at-once” rollback.
Just use canaries, I’m begging you.
I imagine that's got its own risks and challenges if there is a lot of bottlenecks or dependencies, like multi-tenant data store(s).
If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity: whatever latency you have from getting a canary from 0% to 100%, that then determines how quickly you can deploy. The "off" state of a feature flag is usually the pre-existing code, and while nothing is guaranteed to be 100% safe, it's usually a good bet. Then we can worry (post deploy) about ramping that FF from 0% to 100%. It raises the number of steps per deployment, too. (It's not just deploy, it's deploy canary, canary to 10%, canary to 20%, etc., make prod canary. I have seen each of those steps then get bogged down by people needing to have their nerves managed.)
If you mean something like "canary that feature branch" (which I have seen, more rarely) — that style has problems in "how do you keep the feature branch up to date with `main`/`prod`?" (or whatever you call your currently deployed version.) Basically, if I canary a feature branch — and lets say I need to make modifications, and those take time — and then prod is deployed to a new version later than my branch, now that new version has code that, if the user hits my canary, my canary lacks. (And yes, I've seen this in real companies, where a engineer ends up confused as to why their request is failing, b/c it is hitting a canary that is out of date / diverged from prod.)
Yes, that’s the point. I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability.
Note also that I mentioned I am an SRE / DBRE: it’s baffling and frustrating to me that companies will hire people whose job it is to create stable and reliable systems, then ignore them when they say “you’re moving too quickly.” Instead, we get treated as S-tier helpdesk.
Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.
> Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.
I almost mentioned automated stepped rollouts. I guess it's more because there is a staged rollout, non-technical types inexorably inject bureaucracy and manual "approvals" into such things.
I think the other problem I'm faced as an SRE with automated stepped rollouts is "they take too long" and slow down velocity; which, I guess again, you don't see as problematic. I guess I don't know how to "market" that to the SWEs I work with. Even a non-staged k8s Deployment at my last job was frustrating devs due to the time it would take to roll out, even with no real steps, aside from "roll out new version", but the workload itself was so slow that that would still take significant time. (But nobody was willing to invest into "fix workload startup and shutdown times" — some huge fraction of which I think was just loading Google's Vertex library, IIRC, which is … a sign of the times.)
¹ty for your reply, too; that was … damning? skewering?, but insightful.
It's only untested if your intention is to not test it.
There is nothing preventing you from testing a feature flag in e2e tests. In fact, testing is perhaps the primary reason why user overrides are supported by feature flag systems.
Also, since the original comment stated,
> often skips integration/e2e testing
They're probably responding with the expectation of this being compared to doing no testing at all. Merely assuming your FFs are independent, and testing them as such, is greater than nothing at all, even if the rare correlation might exist.
And where a correlation might exist, well, that's what your judgement & expertise is for. Absolutely test that if you find yourself in that case.
You're going way out of your way to imagine problems where there are none. Feature flags are ephemeral and work as an ad-hoc release toggled at runtime. You create a feature flag, you commit the changes you need to commit behind the feature flag. Before switching the feature flag you run tests in preparation for the feature flag release, an finally you flip the flag. After that point, you either rollback the feature flag or you remove the feature flag.
This is not rocket science.