Feature flags in a CI pipeline
andydote.co.uk
andydote.co.uk
- Replacing a job with a newer job for testing purposes, as described in the article.
- Opting out of jobs, like "skip-deploy" or "skip-unit" labels, to speed up CI in some cases when these jobs aren't needed.
- Making some optional jobs required depending on the files that have been modified in the PR. You can get a list of modified files in a PR using the Github API. We use this for example if a new feature necessitates a new CI job but you want this job to be optional for branches that don't have this feature code yet.
1: https://circleci.com/developer/orbs/orb/circleci/continuatio...
The article talks about GH Actions, they have org/repo/env variables.
Even Jenkins has variables usable from the build.
In this case, we're using LaunchDarkly's targeting to serve different variations to groups of users, and also using the in built monitoring LaunchDarkly provides to see how often each is requested etc.
At the end of the day, a feature flag is just a boolean variable determined at run time that can be used to switch code paths on or off. For most purposes that can be as simple as having a list of strings on your user model and checking if it contains a specific value. Adding a service can be valuable if you need to be able to manage those flags more intelligently or monitor them or want to do more structured A/B testing or whatever, but always tying the concept to a commercial service just obfuscates what it actually is and does.
I worked at a company that went far down that slippery slope and now there’s and horribly convoluted bespoke feature flag system that no one enjoys using and no one wants to maintain.
I’ve used LaunchDarkly and it’s really nice. The only downsides I’ve experienced are:
- UI is complicated.
- They have the concept of end users but not end “organizations” (like “set this flag to X for an entire customer organization”). There are hacky ways to do this but they feel icky.
For contrast, adding another service that comes anywhere near touching user data (including user IDs or IP addresses) means doing due dilligence on another service provider, signing a data processing agreement with them, vetting their data protection claims, updating your privacy policy, adjusting your processes for users' data requests (including export, change and removal of their PII) and so on, not to mention adding another recurring expense, point of failure and third-party API dependency.
TINSTAAFL. Of course I understand for many companies (especially outside the EU) a service like this may be much closer to "free" because most companies try to get away with not doing even half of the things I described, whether or not that will come back to bite them. Especially if you're in the business of running a VC-backed startup designed for growth it's often a safer bet to ignore that your engine is on fire as long as you continue to go fast because if you can hit your target (i.e. "exit", whether via acquisition, acquihire or going public) in time that will be someone else's problem or no longer matter.
I have had too many conversations complicated by the fact that when I say off hand "well just use a feature flag for that" or "maybe you should wrap that in a feature flag" they picture massive abstractions and infrastructure and often what I mean is simply "add a configurable variable somewhere, the environment or the user table in a database or a JSON file or something else, then add boring if statements checking that flag". Sometimes it is okay to just start with the basics.
(Again, not to disparage LaunchDarkly. Having "proper" infrastructure for it can provide a lot of nice-to-haves and there are definite benefits to the tools that they value add. Just that sometimes the marketing of tools like that does come at the cost of over-complicating the discussion of the basics of the thing.)
Why not just make them parameters if you are going to put them upfront and turn them into ENVs?
How do you deal with a commit that needs to be rerun? What if the flags have changed in between?
The main problem I see with real feature flags in CI is that it can make reproducing a build nearly impossible. What happens if someone flips a flag in the middle of a build? (where that flag value is checked in multiple places, so the flip happens in between the checks)
It seems impossible to ensure reproducible CI when some of that configuration lives outside of source control. Given the CI files are in source control, you can just change them there, rather than using feature flags. Using feature flags comes with a host of hard problems that you ought to just avoid by not using them.
0 the flag queried is on (true) 1 the flag queried is off (false) 1 an error occurred querying the flag
I think that it would be nice that error and false had different exit codes.
0 - flag on 1 - flag off 2 - error
I'll update the docs soon!
Basically it fires up elasticsearch using docker-compose and then the integration tests run against that. You could use a similar strategy to test different feature flag combinations.
For some of our private projects, we use kts to generate the github action yaml files using this: https://github.com/krzema12/github-workflows-kt
Well worth checking out if you have more complex workflows. Yaml is just horrible in terms of copy paste reuse. Also nice to get some compile time safety and auto complete with our action files.
The tooling around pipelines is awful. A single typo in some variable’s name in a later stage can take minutes to catch. The feedback cycles are very long (cloud machines are much slower than local ones) and IDE tooling is bare-bones.
Just give me one large Python file with some library to manage common actions (building up the job DAG, accessing pull requests, easy shell access, …). We’d have refactoring, Turing completeness, type safety and so much more. A core downside would be managing the complexity of DevOps scripting going berserk. Personally I’d prefer that trade off.
The kotlin scripting support for github actions that I mentioned addresses things like typos, refactoring, and IDE tooling. Try it, it's pretty nice and easy to use. We actually have an integrity check as part of our build that runs the kts script to verify the yaml file stored in the repository is consistent with what the script generates.
I'd love some feedback on what else I could add to this project to make life easier for people.
I'm continuously amazed that none of the major CI providers offer standalone tooling to run and debug your CI pipelines locally. Seems like it'd be a killer feature for anyone working with complex pipelines.