Open to missing something though, curious what others experience has been
The distinction is that you have a different release process, or build a different artifact, from your main codebase. The codebase you are controlling with flags doesn't change when your flags do. This can be done with separate repos if you want one build per repo, but it doesn't have to be.
At this point, your production infrastructure is no longer solely one stateless server + one database, but two databases: your RDBMS and your GitOps repo tracking feature flags. Do you really get enough value from the second GitOps database compared to putting your feature flags in your main RDBMS?
If you're feature flagging client code (i.e. somewhere you don't control rollouts, like mobile and web apps) that adds another layer of complexity.
While it's nice to have a simple system, having built one from scratch and used very mature feature flagging systems, my experience is that production systems hit almost all the edge cases quite quickly and flagging/experimentation systems are forced to evolve quite quickly to actually account for these issues.
Multi-repo or not isn't really an issue. My previous company had flag config in a separate repo, my current company has a monorepo, it doesn't really make a difference.
Modern Postgres scales vertically quite well on modern hardware.
> you still need an audit trail, review processes, etc,
But you need this anyway for the RDBMS in your architecture. You need an audit trail for when engineers need to get into the production database, and to show that their changes passed review, etc. My point is, if you anyway need to build this for your RDBMS, then you can build on top of that for your feature flag system if you throw that into your RDBMS as well.
> my experience is that production systems hit almost all the edge cases quite quickly and flagging/experimentation systems are forced to evolve quite quickly to actually account for these issues.
I think that's more an argument to use a commercial feature flag platform (like LaunchDarkly) instead of a FOSS option. A commerical platform is anyway what I would prefer to recommend! But, with the context of "choose a FOSS option", it seems to me like building on top of RDBMS, rather than GitOps, makes more sense.
This is a complex and nuanced topic, but on my previous team of ~6 where we built a custom solution, we decided against using an RDBMS for multiple reasons, and on my current team where we use the same flagging system across 15 or so >1m requests per second services, there's no way it would work for us. If it works for your use case, that's great! But my advice for anyone else reading would be to put a lot of effort into considering the options as it's hard to change later and has significant impact on how the flagging system is used.
As for whether to use a commercial platform... my preference is probably to build my own with what I need in a system that I can modify as needed, or a commercial platform if there's one ready to go at a good price with the right feature set. I probably wouldn't use an existing open source option here unless I was forking it and treating it as my own from then on, as I find these things need flexibility and customisation. I've yet to see a great open source option.
Flag changes can be pushed directly to the main branch with the correct repo permissions. When using the GitHub UI this involves just a little bit of typing and a few clicks.
>might as well just change the code at that point
If changing the code, running tests, building, and deploying is quicker and less risky then yes that makes more sense.
In these shops, this gets handled via paging on-call engineers. The on-call is sometimes given more latitude if their actions are auditable.
This is nonsense.
The best implementation I've seen was in a Java project. Features where enable or disabled by either the properties file or the database. If a flag was set in the database, then that took precedence. New features would always be rolled out disabled in the properties file. Then in a controlled window the new features would be enabled for a few minutes and logs would be examined. If everything looked good the feature would then be enabled again. After a few days or weeks, the properties file would be updated to have the feature enabled by default and the flag in the database deleted in a later task.
> I'm about to do this thing to mitigate the issue, does it look like the right thing?
It doesn't need to be a code change, can just be a flags change, but if it's a change at all then why not pin it to a commit so that rolling it back is easy and so that the commit sha can be an indicator of which flags are where.
But Git-style version control with history, diffs, branches and pull requests are pretty useful for feature flags and other "app configuration".
Version history and diffs are great for knowing what flag logic changed when + debugging what broke prod.
Branches let you test and preview flag logic changes in your own isolated branch (which you can point the SDK at) — this is a cleaner approach to having a few separate "environments" like development, staging, production which can drift from each other.
Branches are also great for refactoring the schema / structure of all your flags, e.g. deleting a bunch of flags in one go.
Pull requests and approvals are great for when you're making changes to sensitive flags. E.g. you can lock down specific flags.
Pull requests are also great for onboarding nontechnical team members like PMs or sales reps so they can safely make flag changes themselves but require approval from an engineer (at least while they learn to use the system). Empowering nontechnical people is also why a UI is important.
Branching and pull requests are also a great way to prevent conflicts / overwriting other team members flag changes.
So Git-style features are pretty useful, but you also want the UI and you only want to enforce pull requests for specific flags or team members — this is what we built at Hypertune.
So instead of controlling flags from a website, you get the benefits of git merge, PR's, reviews, documentation etc without having to rebuild it.
I like the concept since it brings accountability. But it's just a need that larger orgs have, but by that point have likely internally built a flag system and so transitioning is difficult.
I’m not sure I have a comprehensive solution. I just know which ones I hate more than which others. The repo is the least obnoxious of the options.
For systems of sufficient scale, it's fairly standard to keep flag changes outside of git so that they can be flipped without a pr. That way the flag change UI can apply other validation steps before any change is attempted such as ensuring valid enrolment ranges (no accidental overlap, and no accidental rollouts to 100% instead of 10%), and the associated rollout analytics can be shown alongside the changes.
You can also override things in emergencies more easily, which is the parent's point.
Though you could also create your own audit system (just make sure it functions even when the entire site is down)
In particular, a change to the yaml for feature flags could bypass most of your build and test pipeline, and changes could be deployed more quickly.
OTOH, you still need to figure out a deployment strategy for it.
1. The time to delivery is potentially much much shorter.
2. There's a built-in rules engine for targeting. You could integrate this in! But it feels nice having it separate.