How Growth Hackers Get A/B Testing Wrong | Hacker News Reader

How Growth Hackers Get A/B Testing Wrong

blog.klaviyo.com

8 pointshemancuso6 comments

You have to wonder if its even possible for the same tools that you use to A/B test small changes (i.e. in UI, menus, branding language) can also be applied for these more broad systemic changes. What it sounds like a lot of the software mentioned in this article does is to maximize locally. Is it even possible to then use the same process to maximize globally? If the answer is yes, then companies should ask themselves if they're willing or able to make such large systemic changes on such little data. I wonder how many people would be able to make such game changing decisions off of the results of these A/B testing surveys. My guess is probably not a lot.

I think it probably is, it just takes more planning. Take email A/B testing - if email is a main driver of your usage activity, you could split your users or list into two groups and then send them different emails over a period of months. It'd be slightly more manual (you'd have to segment the list) and then leave one group off for a few months, but it would let you easily gather very significant data.

> You have to wonder if its even possible for the same tools that you use to A/B test small changes (i.e. in UI, menus, branding language) can also be applied for these more broad systemic changes.

No need to wonder; the answer is yes. For example, Optimizely[0] lets you test different pages (redirect tests) against one another, so instead of just testing button locations you can test different offers, pricing options, branding, and other "systemic changes." You can also run multi-page tests, in which the variation being tested can span across many pages.

[0] https://www.optimizely.com

(I have no affiliation with Optimizely, although I'm certified in it and use it frequently in my consulting.)

Off-topic: there isn't a single tech term that has been such a turn-off for me as the whole 'Growth Hacker' thing.

At first I liked the term, because it finally gave me an easy way to describe my skills other than "a marketer who can code with a focus on metrics"... Then as I started reading forums and articles by and for "growth hackers," I found the content so vapid and useless that I now keep far away from that term and community.

Yes, for the love of god, just call it marketing. All marketing is supposed to be data driven anyway. We don't need a new buzzword to describe a marketer behind a computer.

Reply on news.ycombinator.com