Hacker News Reader

Comment by treis | Hacker News Reader

>Instead try to improve the customer experience, make better products, improve customer service.

Without a metric to say what is "better" and a method to measure it this is empty advice.

There are many other metrics that do not involve AB testing. You can just survey customer experience before, during and after a purchase for example. I never said to throw out all metrics.

With AB testing your are optimising for a specific outcome. Usually higher conversion. As pointed out in the article eventually you'll end up with a bunch of colourful buttons and scary texts that persuade the user to click. A lot of the "only 2 seats/rooms available" are lies to scare the user into a conversion.

But does AB testing provide the only or even best metric for that? It probably is the cheapest way requiring the least engagement with the lab mice.

AB testing is how you isolate a change and measure the impact. It's the only real way to be able to associate cause and effect. Best you can do otherwise is measure something over time while making changes. You can try and correlate changes with outcomes but it's hard to be sure the change is what drove the outcome.

That sounds pretty accurate. Anyone who's gone through some econometric classes would know that split testing is (almost) the only way to get to (close to) causality - all the rest is black magic called correlation which can also lead you to the conclusion that Nicolas Cage's movies lead to people drowning:

https://www.thenationalnews.com/uae/nicolas-cage-movies-link....

https://www.wnycstudios.org/podcasts/otm/articles/spurious-c...

Reply on news.ycombinator.com