Hacker News Reader

Comment by beeon | Hacker News Reader

Maybe the industry you work in is relevant here. In e-commerce, A/B testing the position and color of "add to cart" button can yield legitimate revenue multipliers. People's opinions are irrelevant in that kind of A/B test, all that matters is the likelihood they continue down the funnel.

View on HN

This is always the straw man A/B testing people reach for. When in reality, this only works on the worst possible designs or bizarre layouts, that no one really uses. It's a myth that you can somehow ring out extra few % by making tiny changes to fonts and colors -- in every case, they simply stop the experiment when the results tip in their favor.

The ONLY time I've ever seen it used successfully was to make changes to the layout of ads, making them look like more organic content or likely to be accidentally clicked. You can see this at work if you've ever clicked on an ad by mistake, or maybe you were trying to close the ad and noticed someone has "optimized" the close button placement using one of these A/B tools.

There it is, that's the market for these tools. That represents the vast majority of the these companies use-cases, revenue and usage. Anyone else who implies an innocent intent is either ignorant or inexperienced.

Experimentation is a toolkit for optimizing things. That's it. For example, if you are Uber, you might offer a bonus to people to work more. How much money do you offer? Is there a relationship between how much you pay and how well people perform? To determine this, you need empirical data, you need to be able to split people into groups, and you need to be able to measure things about the groups. And you need some guardrails to keep yourself from p-hacking. As the number of people and the amount of money increases, the number of interesting questions rises rapidly. These are all things you get from an experimentation platform.

Experimentation works best when a) it's easy to try different parameters for a thing, and b) there is a lot of money involved.

Your comment is weird to me because you seem to be implying that experimentation is a cult because there are no things worth optimizing, and/or optimizing things has a negative societal effect. Those things are obviously true some of the time, but it's not very interesting.

Reply on news.ycombinator.com