Hacker News Reader

Comment by mtrimpe | Hacker News Reader

Really love the animation, but I can't help but wonder if A/B testing couldn't be explained more powerfully.

Within the example of the store for example, the real pain A/B testing addresses is cyclical variation leading to incorrect conclusions.

Working from that you could've shown the story of:

* the store owner rearranging his display and seeing sales go up

* but then the rest of the year sales actually going way down

* and when he changes things back to the old layout sales went waaay up again! How can that be?

* Cue image of cyclical charts and the facepalm of realizing it wasn't his redesign at all!

* Now ... on his website he can send visitors to different versions at the same time!

* So he'll never have a year of bad sales again!

View on HN

Those are good points, but we were gunning for the marketer who isn't familiar with A/B testing. This because we see that a lot of our potential customers have usually heard the term somewhere but need a basic, easy-to-understand explanation.

Do you think it achieves that purpose?

Yeah, you could interpret my feedback as saying that you're addressing the what but not the why.

Looking at it as sales/marketing copy though I'd say that your biggest competitor by far isn't people not understanding what it is but people thinking "Why don't I just change my site and see what happens?"

In that context I'd guess this story telling format is perfect for taking a concept like 'prevent acceptance of false hypotheses by controlling for external variation through randomized trials' and turning it into a visceral "Oh fuck. That's painful!" moment your target market can relate to.

Yes, we indeed missed explaining why A/B simultaneously is how it should be done v/s A after B. Thanks for the inputs.

Perhaps you should deploy an alternate version, and meassure conversion rates? ;-)

Seriously, I think you make a great point of selling your service. I also think you gloss over things like cyclic variations -- but I'm not sure it's your job to educate people in the subtle art of fact oriented market research (or what one should call it).

For those that don't have the time to learn, changing it probably wouldn't matter. For those that do -- chances are they probably will find it out from a more comprehensive resource (like, say, a good book).

Exactly. For someone who's just starting with A/B testing, this serves as a good/interesting introduction. But for someone who really wants to understand the detailed behind-the-scenes, this will obviously fall short.

Coming to your suggestion, do you have a good book/website in mind?

I'm afraid I don't have any books to recommend -- maybe someone else has?

> (like, say, a good book).

Can you suggest one?

Yes I agree that it feels that the animation is not powerful enough.

Reply on news.ycombinator.com