github.com
Full disclosure: made by a former LaunchDarkly employee + current fan.
Open to missing something though, curious what others experience has been
So instead of controlling flags from a website, you get the benefits of git merge, PR's, reviews, documentation etc without having to rebuild it.
I like the concept since it brings accountability. But it's just a need that larger orgs have, but by that point have likely internally built a flag system and so transitioning is difficult.
I’m not sure I have a comprehensive solution. I just know which ones I hate more than which others. The repo is the least obnoxious of the options.
Flag changes can be pushed directly to the main branch with the correct repo permissions. When using the GitHub UI this involves just a little bit of typing and a few clicks.
>might as well just change the code at that point
If changing the code, running tests, building, and deploying is quicker and less risky then yes that makes more sense.
In these shops, this gets handled via paging on-call engineers. The on-call is sometimes given more latitude if their actions are auditable.
This is nonsense.
> I'm about to do this thing to mitigate the issue, does it look like the right thing?
It doesn't need to be a code change, can just be a flags change, but if it's a change at all then why not pin it to a commit so that rolling it back is easy and so that the commit sha can be an indicator of which flags are where.
For systems of sufficient scale, it's fairly standard to keep flag changes outside of git so that they can be flipped without a pr. That way the flag change UI can apply other validation steps before any change is attempted such as ensuring valid enrolment ranges (no accidental overlap, and no accidental rollouts to 100% instead of 10%), and the associated rollout analytics can be shown alongside the changes.
You can also override things in emergencies more easily, which is the parent's point.
Though you could also create your own audit system (just make sure it functions even when the entire site is down)
1. The time to delivery is potentially much much shorter.
2. There's a built-in rules engine for targeting. You could integrate this in! But it feels nice having it separate.
The distinction is that you have a different release process, or build a different artifact, from your main codebase. The codebase you are controlling with flags doesn't change when your flags do. This can be done with separate repos if you want one build per repo, but it doesn't have to be.
At this point, your production infrastructure is no longer solely one stateless server + one database, but two databases: your RDBMS and your GitOps repo tracking feature flags. Do you really get enough value from the second GitOps database compared to putting your feature flags in your main RDBMS?
If you're feature flagging client code (i.e. somewhere you don't control rollouts, like mobile and web apps) that adds another layer of complexity.
While it's nice to have a simple system, having built one from scratch and used very mature feature flagging systems, my experience is that production systems hit almost all the edge cases quite quickly and flagging/experimentation systems are forced to evolve quite quickly to actually account for these issues.
Multi-repo or not isn't really an issue. My previous company had flag config in a separate repo, my current company has a monorepo, it doesn't really make a difference.
Modern Postgres scales vertically quite well on modern hardware.
> you still need an audit trail, review processes, etc,
But you need this anyway for the RDBMS in your architecture. You need an audit trail for when engineers need to get into the production database, and to show that their changes passed review, etc. My point is, if you anyway need to build this for your RDBMS, then you can build on top of that for your feature flag system if you throw that into your RDBMS as well.
> my experience is that production systems hit almost all the edge cases quite quickly and flagging/experimentation systems are forced to evolve quite quickly to actually account for these issues.
I think that's more an argument to use a commercial feature flag platform (like LaunchDarkly) instead of a FOSS option. A commerical platform is anyway what I would prefer to recommend! But, with the context of "choose a FOSS option", it seems to me like building on top of RDBMS, rather than GitOps, makes more sense.
This is a complex and nuanced topic, but on my previous team of ~6 where we built a custom solution, we decided against using an RDBMS for multiple reasons, and on my current team where we use the same flagging system across 15 or so >1m requests per second services, there's no way it would work for us. If it works for your use case, that's great! But my advice for anyone else reading would be to put a lot of effort into considering the options as it's hard to change later and has significant impact on how the flagging system is used.
As for whether to use a commercial platform... my preference is probably to build my own with what I need in a system that I can modify as needed, or a commercial platform if there's one ready to go at a good price with the right feature set. I probably wouldn't use an existing open source option here unless I was forking it and treating it as my own from then on, as I find these things need flexibility and customisation. I've yet to see a great open source option.
In particular, a change to the yaml for feature flags could bypass most of your build and test pipeline, and changes could be deployed more quickly.
OTOH, you still need to figure out a deployment strategy for it.
The best implementation I've seen was in a Java project. Features where enable or disabled by either the properties file or the database. If a flag was set in the database, then that took precedence. New features would always be rolled out disabled in the properties file. Then in a controlled window the new features would be enabled for a few minutes and logs would be examined. If everything looked good the feature would then be enabled again. After a few days or weeks, the properties file would be updated to have the feature enabled by default and the flag in the database deleted in a later task.
But Git-style version control with history, diffs, branches and pull requests are pretty useful for feature flags and other "app configuration".
Version history and diffs are great for knowing what flag logic changed when + debugging what broke prod.
Branches let you test and preview flag logic changes in your own isolated branch (which you can point the SDK at) — this is a cleaner approach to having a few separate "environments" like development, staging, production which can drift from each other.
Branches are also great for refactoring the schema / structure of all your flags, e.g. deleting a bunch of flags in one go.
Pull requests and approvals are great for when you're making changes to sensitive flags. E.g. you can lock down specific flags.
Pull requests are also great for onboarding nontechnical team members like PMs or sales reps so they can safely make flag changes themselves but require approval from an engineer (at least while they learn to use the system). Empowering nontechnical people is also why a UI is important.
Branching and pull requests are also a great way to prevent conflicts / overwriting other team members flag changes.
So Git-style features are pretty useful, but you also want the UI and you only want to enforce pull requests for specific flags or team members — this is what we built at Hypertune.
As you add more of them though, they add tech debt and make the code harder to reason about. Developers are rarely motivated to clean them up after rollout.
It's stakeholders not giving engineers necessary time for upkeep, probably caused by engineers lacking ownership, or failing to communicate
At the end of the day it’s still fuckery I was unable to avoid.
The best companies I've been at where those where product and engineering really operate as one team.
In Limited Work In Progress (LWIP aka WIP-Limited) processes, if you are blocked on your story you help someone else get unblocked on theirs. And then escalate if there is no movement.
Admittedly, there’s still a gap there where a toggle may be needed until the end of an initiative (an Epic in Scrum terms) but that’s going to be at least an order of magnitude, and likely 2 orders, fewer toggles to deal with.
When the order of magnitude changes so does the solution.
Been doing that with my new hires in other jobs.
We already run a giant pool of relays for our apps to connect to, both to save ingress/egress costs, and to provide a backup if LD blips out for a while.
[Edit: yep, it actually uses the Relay Proxy, in offline mode. See https://github.com/dorklyorg/dorkly/wiki/5.-Architecture]
In fact, their entire API for flag manipulation and definition (essentially all the operations you can do with their UI, but programmatically) is open too: https://app.launchdarkly.com/api/v2/openapi.json
One could imagine running an OpenAPI → Go/Java/Rust/Python/etc. code generator on their OpenAPI spec, and then gradually implementing their entire API as open source.
Not sure how they would feel about that… although I suspect more usage of an LD-compatible API would only be good for them.
Of course, it makes sense for all those things to be open. The majority of smaller startups they're competing against will have settled on OpenFeature. DataDog too releases some equivalently surprising things as open source: eg. https://vector.dev/
If this is a limitation for anyone I'm happy to work with you on modifying the terraform module to support HA.
Which yaml files is one supposed to edit to actually modify the flags?
This is good feedback. I'll clarify the comments
The flags are really useful for things like enabling just a fraction of traffic, or ensuring you can switch a feature off much quicker than a full deploy would take.
be happy to walk you through how we use it. Shoot me an email if you want to chat. wayne at inaccord.com
We almost never shared flags across our fairly chunky servicees. We usually found a softer way to do it.
Even with Consul, you can still have enough skew that a few requests in the middle might see A and !A if your request rate is high enough. So it depends on your business model and architecture if that’s acceptable.
In most such cases you have several instances of your backend running in parallel for scaling and redundancy and when making a release, instances of several versions run concurrently. So you don't have a "atomic upgrade" available
With multiple services coordinating upgrade is even harder.
Patterns like expand-contract helps you manage this..E.g. first add the new endpoint to server, then move clients over, then remove old endpoint.
So..feature flags is just a way of dragging this process over longer time period, and roll over % of traffic. Instead of coupling changes to service releases you roll over using config.
Used them a lot in backend to backend, backend to DB etc scenarios, has been hugely useful to us and would never work without it.
But, of course depend on context what you are doing.
... though there are only three officially-supported OpenFeature adapters so far (Java, NodeJS, .NET)
Probably answering my own question, but the main reason I can think of would be if your app doesn't have some kind of business admin panel capability.
I suppose my bias is also that in my sphere of web dev, we build these business panels pretty frequently, so feature flags and a UI for toggling them is something that makes sense to do within the app rather than add a third party service (and associated cost + potential latency) for it.
However, there are settings that apply to all of them in our library code. For those, we use feature flags, and they are loaded from a network service, environment variables, code and config files. The overriding logic is complicated for legacy reasons and prone to bugs, so we are moving to a centralized feature flag system.
Thankfully GitHub has an easy web-based Pull Request process. It's not as simple as a custom UI, but could be used by non-engineers to change flags.
That's a licensing thing, not a feature flag.
But there's also a surprisingly common situation with big customers who want to limit and control certain UI updates so they can ensure that employees are trained appropriately.
But it's all the same code under the covers. The functionality of what is enabled encoded into a license key - that's the same type of functionality as "does this deployment allow people to sign up by email" or "has this deployment enabled the super secret power user test functionality?"
But that's all under "if dataSource.toggle then {something}"
Unaudited, untracked web UI based feature flagging has been my peeve with the other solutions. I'm surprised all the feature flagging solutions out there don't have this as a default. The next step would be to use an artifact versioning repository to decouple from S3.
Checking now they have about 4 million subscribers. Probably not gonna be great for SEO longterm.
also can use GitHub
https://github.com/theogravity/feature-manager-wrapper
Edit: It looks like it's a backend replacement to LaunchDarkly, but you can still use the LD client from what I'm reading here, so there's nothing for me to integrate here.
For instance, here are is the repo for contributed Providers in Go: https://github.com/open-feature/go-sdk-contrib/tree/main/pro...
And here is LaunchDarkly's official Java OpenFeature provider: https://github.com/launchdarkly/openfeature-java-server