Patching around a C++ crash with a little bit of Lua
rachelbythebay.com
rachelbythebay.com
At the very least extremely liberal use of feature flags is table stakes for helping ops teams stay a bit sane.
Restrained use of feature flags is fine, because if you only have 2-3 flags, you have a decent chance of testing a useful subset of possible combinations. I've seen projects with 20+ flags even after instituting a policy that flags can only be in the code for X months. They were a nightmare in production. It was more common to roll back to "previous known good version + flags" than to roll back flags alone, which notably defeats the point of feature flags because they're coupled to the version anyway.
Even "push on green" practices are actively poisoned by feature flags because green isn't representative of prod. Even if every individual feature is thoroughly tested, that's usually in isolation, not in combination with other features. The most accurate testing would be the set of flags that's meant to be used in production, but that brings us back to the same problem; as soon as even one flag changes, the testing was no longer representative of prod, and if nobody ever changes any flags, then the feature flags were at best useless.
I'll take feature flags over diverging branches, but I'd still much rather take true converged development over rampant feature flags.
You add and test the feature flag before you deploy in to prod so you can disable the feature quickly if shit hits the fan.
Same for removal, disable first. Make sure it's all ok, then remove the feature later.
Of course in the original article there's not much to be said, and I would have probably shipped the "handle the new header" functionality on the server as-is. Feature flags wouldn't really help! But there are so many things where feature flags have been helpful.
The main point is that you want feature flags to help control usage, but really new features that would use them should do their best to not look at the flag inside of the business logic, because that's the stuff that's brittle.
Or even, since the bug is in some C++, we can try "\0-Can-Do-Fancy-Compression" and maybe the C++ code thinks this header is empty because apparently it's still the 1970s where Bjarne Stroustrup lives.
Would that guess definitely be correct? No. Is it a reasonable guess for code which is apparently untested and buggy? I think so.
Because C++ loves implicit conversion this works even when the API didn't intend char* because it just calls the constructor on std::string to make one from the pointer. "Hooray"
I'm trying to figure out what profound excuse the client has for not using gzip et al; "feature phone land" definitely provides no shortage of possibilities (hence the morbid "oh no I must know").
That's a hallucinatory rabbithole I could get stuck staring down all day because it's too fun (welp hehe) and there's surely a simple explanation I'm missing :)
I once worked at a company where someone pushed a release that added timestamp based cache busters to every js/css file. We used a similar approach to just drop that query parameter at the edge, which allowed us to get cache hits and stop serving every single asset from object storage.
So less of a hack, and more of a "hey, that's the way it works" - or - Clark's third law is highly dependent on your POV and aptitude.
Downside to quick workarounds like this is that they almost always end up becoming land mines. No one who looks at that config is going to know why it looks that way and will be tempted to take it out.