Hacker News Reader

Comment by rmetzler | Hacker News Reader

If I read the first mistake correctly, then getFeatureFlag() has the side-effect to count how often it was called and uses this to calculate the outcome of the experiment? Wow. I don't know what to say....

View on HN

Yeah I felt that way too. Initially I thought I wasn't sure what I was missing, since the only difference is that the order of the checks is switched, and the function will still return the same true/false in both cases. Then I thought about side effects and it felt icky.

That's how every one of these tools works, that’s the whole point of using them: you only call them when you’re going to actually show the variation to the user. If you’re running a test that modifies the homepage only, you shouldn’t be calling that decision method in, say, your global navigation code that you show everyone. Or, for instance, if your test only affects how the header looks for subscribers, you have to put an outer if statement “if subscriber“ before the “if test variation.“ How else would it correctly know exactly who saw the test?

This is indeed the case. Have run into a few surprising things like this when implementing posthog experiments recently

When you call the feature flag, it’s going to put the user into one of the groups. The article is saying you don’t want to add irrelevant users (in the example, ones that had already done the action they were testing) because it’s going to skew your results.

The point is from an api design perspective something like

"posthog.getFeatureFlag('experiment-key')"

doesn't look like it's actually performing a mutation.

Writing an article about developer mistakes is easier than redesigning your rubbish API though.

Yep gross...

Reply on news.ycombinator.com