Good Experiment, Bad Experiment

reforge.com

25 points·xtacy··4 comments

This is philosophically interesting, but little of this really is about _running_ a good experiment. Perhaps it's about choosing which ones to run?

Kohavi's book will probably provide much more value than this kind of abstract post. See https://experimentguide.com/ for more details.

This bit is crucial for running a good experiment:

> Good experiments use tight exposure groups

Not all your users will have the conditions necessary for the experimental treatment.

You should only look at changes in behavior where the treatment condition was true. Otherwise your experimental effect is diluted.

But your comparison between experiment and control groups must be neutral on the condition, that is, users in the control group would have seen the treatment if they were in the experiment group (counterfactual).

> Good experiments define success up-front.

This is absolutely critical. If you're not defining success up-front, you're not running an experiment. You're just doing a staged rollout. You can use the data to craft whatever story you want for most changes.

tl;dr former Slack director of product outlines twelve principles for a/b type experiments with features, products, whatever. (ex Good experiments involve analyzing the results as opposed to 'ship/kill?') Each principle has a couple of sentences of elaboration and calls out the opposite style to avoid.

Reply on news.ycombinator.com