Hacker News Reader

The biggest issue with those three hypotheses is one of them, the noticing the button, almost certainly isn't being tested. But, how the test goes will inform how people think about that hypothesis.

View on HN

Rate of traffic on the checkout page, divided by overall traffic.

We see a lot of ghosts in A/B testing because we are loosey goosey about our denominators. Mathematicians apparently hate it when we do that.

That doesn't test noticing the button, that tests clicking the button. If the color changes it is possible that fewer people notice it but are more likely to click in a way that increases total traffic. Or more people notice it but are less likely to click in a way that reduces traffic.

Good observation that the noticing doesn’t get tested.

Would there be any benefit from knowing the notice rate though? After all, the intended outcome is increased sales by clicking.

This is what I was driving at in my original comment - the intermediary steps are not of interest (from the POV of the hypothesis/overall experiment), so why mention them at all.

Probably not, but then that hypothesis should not be part of the experiment.

Reply on news.ycombinator.com