Hacker News Reader

Comment by citrablue | Hacker News Reader

What you said is correct, but I'd like to point out that controlled scientific experiments may not be the right approach for e.g. optimizing conversions on a website.

The reason is that websites are a dynamic environment. All things equal, better controlled experiments are great. However, visitor behavior, especially when from dynamic sources (google serps change weekly), changes all the time.

And that's why I prefer MAB over A/B tests -- A/B tests don't adapt to a dynamic system, so we often wish away the changes in the system to use it. Does anyone go back and re-test their biggest wins?

View on HN

> Does anyone go back and re-test their biggest wins?

Yes, absolutely! We do research first, then come up with simple, well-controlled tests. Once we have a winner we can either lock it in, but often we continue to research and experiment on the new knowledge we gained. A hefty minority of the tests I implement build on past wins to further flesh out what works and what doesn't with knowledge and the data to back it up.

Very interesting. It seems to me that doing incremental work like this might end up in a local minima/maxima. Do you have any advice on how to avoid pitfalls like that? Are you testing radically different ideas along with your incremental improvements?

From a workflow perspective, MAB is a bit difficult to find radical improvements from. The radical improvements come from fundamental design changes, of which it would be very expensive to create a bunch of radically different variants.

MAB is best used where you can generate a bunch of variants cheaply and hope for a 30% gain.

Cool, thanks.

How do you do knowledge transfer when you're documenting what you've learned? I have found even simple changes often don't reproduce on the same site. It feels like a easy trap for an expert practitioner to fall into -- the belief they can predict the outcome of a test based on a different test they ran.

Sometimes, yes, but often it's not for the reasons people think. This seems very much in Danny Kahneman land -- we can't trust what our brain is telling us, that this experience will have similar results in this other context.

I recently worked at a place that ran > 20 websites, huge traffic numbers, with around 100 simultaneous tests. We would poll each employee about their guess for the test winner. The results were about on par with everyone randomly guessing.

Reply on news.ycombinator.com