You probably don’t need A/B testing | Hacker News Reader

You probably don’t need A/B testing

oliverpalmer.com

Lets fix the title - if you think you need A/B testing, you don't. I have explained this to small organizations that are running sites getting tens of thousands of hits and just completely floundered at it.

The people whom A/B testing can help are running sites with million of hits.

Haha, I like your title but I don't think it's accurate.

Many, many sites with millions of visitors are not yet A/B testing. Vastly more than you would think.

Ok, lets take it a step further ;)

The sites with millions of hits not A/B testing might be helped by an A/B test but more likely will just chase ghosts and give it up.

Or they could just hire a good UI/UX and conversion expert and get the insights for free.

Take agriculture - you are running a large farm which is hundreds to thousands of acres. Do you run experiments with soil, seeds, fertilizer, crop rotation schemes to increase yield? No, that would be silly, take forever and make you poorer and perhaps no wiser. Instead, you hire a "expert" who has done it before and knows the "best practices".

Very interesting analogy but not a good equivalence IMO. Soil and rotation are pretty standard in that the best permutations are already known, unless you want to experiment something new. Websites do not have such known standards- a crappy looking amazon website works better than most UI experts would suggest.

Strongly disagree - A/B testing can be heavily used in e-commerce and online shops. You don't need "millions" of hits in order to test which products do better on the homepage, which discounts encourage more people to click buy, etc.

UX research can act as observations to form an initial hypothesis, while A/B testing can acquire the necessary data to perform experiments in way that assimilates the scientific method.

It might be a scale issue for sure, but qualitative data alone is not enough, especially when dealing with user biases and small, non representative populations.

Not enough for what?

To solve a customer pain or problem

Disagree - thousands of hits can provide huge amounts of insightful data to make A/B testing valuable and productive when its captured and analyse effectively as part of a broader UX and Design System strategy. If you don’t have those it doesn’t matter what volume of data you have.

Using data to test product iteractions, interface changes and catch low hanging fruit is totally standard and widely practiced. Companies not willing to use data driven measurement are lagging behind by at least 5 years.

Reply on news.ycombinator.com