As an SRE / DBRE, I hate FFs because it means I can be lulled into a false sense of security when something I saw didn’t initially fail, only to start doing so the next week.
Just use canaries, I’m begging you.
Just use canaries, I’m begging you.
If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity: whatever latency you have from getting a canary from 0% to 100%, that then determines how quickly you can deploy. The "off" state of a feature flag is usually the pre-existing code, and while nothing is guaranteed to be 100% safe, it's usually a good bet. Then we can worry (post deploy) about ramping that FF from 0% to 100%. It raises the number of steps per deployment, too. (It's not just deploy, it's deploy canary, canary to 10%, canary to 20%, etc., make prod canary. I have seen each of those steps then get bogged down by people needing to have their nerves managed.)
If you mean something like "canary that feature branch" (which I have seen, more rarely) — that style has problems in "how do you keep the feature branch up to date with `main`/`prod`?" (or whatever you call your currently deployed version.) Basically, if I canary a feature branch — and lets say I need to make modifications, and those take time — and then prod is deployed to a new version later than my branch, now that new version has code that, if the user hits my canary, my canary lacks. (And yes, I've seen this in real companies, where a engineer ends up confused as to why their request is failing, b/c it is hitting a canary that is out of date / diverged from prod.)
Yes, that’s the point. I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability.
Note also that I mentioned I am an SRE / DBRE: it’s baffling and frustrating to me that companies will hire people whose job it is to create stable and reliable systems, then ignore them when they say “you’re moving too quickly.” Instead, we get treated as S-tier helpdesk.
Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.
> Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.
I almost mentioned automated stepped rollouts. I guess it's more because there is a staged rollout, non-technical types inexorably inject bureaucracy and manual "approvals" into such things.
I think the other problem I'm faced as an SRE with automated stepped rollouts is "they take too long" and slow down velocity; which, I guess again, you don't see as problematic. I guess I don't know how to "market" that to the SWEs I work with. Even a non-staged k8s Deployment at my last job was frustrating devs due to the time it would take to roll out, even with no real steps, aside from "roll out new version", but the workload itself was so slow that that would still take significant time. (But nobody was willing to invest into "fix workload startup and shutdown times" — some huge fraction of which I think was just loading Google's Vertex library, IIRC, which is … a sign of the times.)
¹ty for your reply, too; that was … damning? skewering?, but insightful.
I imagine that's got its own risks and challenges if there is a lot of bottlenecks or dependencies, like multi-tenant data store(s).