Hacker News Reader

Comment by spiderfarmer | Hacker News Reader

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird.

For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better.

But how many websites / apps really have >100K uniques? If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc.

And if you have <100K uniques, does the increase in conversion pay for the amount of time you have invested in setting up the test?

In my experience, only when you test some completely different pages you'll see some significant differences.

View on HN

You are right, and the guy that responded with a test of just under 150 conversions is a great example of exactly what people get wrong.

I have been doing design optimization for nearly a decade now and virtually every example I've seen made public is incredibly poor. Sometimes they do not even include any numbers, but just say, look first I had a 1% conversion rate and now I got 1.2%! I wish I had a good example of optimization case study, but I don't recall the last time I saw one.

Without giving anything away, I have done tests this year which required over a million impressions to find out in a statistically significant way not to make any changes.

My general theory is that a never optimized design with a confusing UI has a lot of low hanging fruit. You start cleaning up the bad elements and conversion rates can double or triple. Even if the sample size is less than ideal, these really big pops will be apparent.

Design knowledge is better now than it was in 2003. Mobile forces designers to use one column and leave out a lot of crap. There are a lot of good examples that get copied and good out of the box software. That means when you start optimizing, the low hanging fruit is gone and you need really big sample sizes. Once the low hanging fruit is gone, often those big samples just tell you not to make any changes.

Thinking about free-to-play mobile games recently (an area I have no experience in.) If 1% or fewer of users are converting you really do need a huge install base to beat your competitors at optimization. You need millions of users just to get to 20,000 or 30,000 paying players to test behavioral changes on. That means there actually is some staying power for the winners, at least for a while.

My professional advice to people who make these unmeasurable changes is: Then don't do them. The worst part about unmeasurable changes is you can't verify if it has a negative effect either and you're essentially saying "I'm focusing on making this change when it will have negligible impact on the business."

You have better things to do.

I agree, every article about A/B testing is talking about making small changes and measure them, while you should be measuring only big changes or nothing at all.

People read about Google testing different shades of blue on their home page and want to do the same on their website.

They tend to forget that Google has a billion unique users per day, and their website a few hundreds.

> the difference in conversion rate is not measurable

Wut?

Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m

That's 26% sales increase from changing a button's color.

If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll futz with things that make no difference most of the time and eventually give up. Experimentation is not about testing random permutations (unless you've got an infinite amount of time). It's about coming up with a reasonable hypothesis and then testing it. I study a web page until I come to a conclusion in my mind that something could really use some improvement. Then, even though I'm sure of it in my mind, I test it, because "sure of it in my mind" is wrong about half the time.

One note to those who use Optimizely: you do need to wait at least twice as long as Optimizely thinks, because statistical significance can bounce around with too little traffic. Optimizely thought the above test was done at about 2000 views, which was far too little results to be conclusive, with only ~20 sales.

You really should question the statistical significance of those numbers.

Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good.

So: probably better, but should have run it longer.

The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand...

I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained the same for the entire test, rather than there being a specific period where "yellow" excelled.

> I wanted Optimizely to say it was 100% significant

That's not how statistical significance works...

You should contact Optimizely and let them know right now.

https://www.optimizely.com/contact/

They'll need to reeducate their statisticians right away!

Optimizely did an A/B test that showed that customers respond better to rounded-up numbers. ;-) the real world is messy.

Years ago my team's statistician did a competitive review of various AB test apps, and reported various ways in which the UIs make statistically invalid statements to the user.

They probably do, but why would they want to do so, when doing it would make their results appear less conclusive?

The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about the size of the differences. It also can't be 100. The normal distribution has an infinite range, so a finite limit can never capture 100% of it.

Optimizely is in a rough spot. People don't like having to think through experimental design, and they are really, really bad at reasoning about p-values. To try to fix the people part, they came out with the sequential stopping rule stuff (their "stats engine), but they never really published much justifying it. The other alternative would be to move the experiments into a Bayesian framework, but that has a lot of it's own problems. When they acquired Synference, that was one of the likely directions to take (along with offering bandits), but that didn't work out and those guys have since left.

> but they never really published much justifying it

Not that I'm trying to defend Optimizely (I'm not a huge fan, but for other reasons...).

I can't vouch for the quality either, but they did publish something about it[0] - that at least looks quite scientific. Happy to read any critique of course.

[0] http://pages.optimizely.com/rs/optimizely/images/stats_engin...

Latex is a wonderful way to make a marketing paper look like a scientific one. It doesn't accurately describe the method, but that isn't really its purpose. It's a more technical description of the blog post, meant for people using the product to understand some of the tradeoffs and get more accurate results.

They are still having people make very fundamentally flawed assumptions about the data, which results in incorrect conclusions, and they are still not presenting the results in a way that people correctly interpret them. That being said, those are really hard to solve, and models that would try to correct for them would likely require a lot more data and be overly conservative for more people.

What are your reasons for disliking Optimizely?

It may be marketing but I was able to implement a sequential A/B test based on it. Admittedly, I did need to do some work beyond merely copy/pasting an algorithm, but all I really needed to do was read their paper and some citations. I do believe that this document does describe a viable frequentist test and my implementation of it worked pretty well.

Disclaimer: I do stats work at VWO, an Optimizely competitor.

(Also if you want to read our tech paper, here it is: https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic... This describes our Bayesian approach, which we believe to be less likely to be wrongly interpreted by non-statisticians.)

Hi, this is Leo, Optimizely's statistician. If you're looking for a more scientific paper, maybe take a look at this one we wrote recently: http://arxiv.org/abs/1512.04922

Should have everything you would ever want to know about the method.

I agree with you that the problem of inference and interpretation between A/B data, algorithms, and the people who make decisions from them is a hard one and worth working on.

That said, I do think the two sources of error our stats engine addresses - repeatedly checking results, and cherry picking from many metrics and variations - did make progress in having folks correctly interpret A/B Tests. This did result in more conservative results, but the benefit was that the variations that do become significant are more trustworthy. I think this was absolutely the right tradeoff to make for our customers, and trustworthyness is a pretty important aspiration for stats/ML/data science in general.

Of course I did write the thing, so I'm not very impartial.

I was tempted to make a snarky comment about using LaTeX, but I'm not sure it's entirely fair. It doesn't seem like just a bunch of MarketingSpeak wrapped in LaTeX to be honest.

My issue with Optimizely are mainly how they essentially ditched us as (paying) customers. We were admittedly small-fish, but we were paying and were willing to pay more, but they switched to Enterprise-vs-Free without any middle grounds. Enterprise was way too expensive for us. Free didn't include essential features, so we were just stuck.

I ended up writing an open-source javascript A/B test client[0] (and recently also an AWS-lambda backend[1]), but it still has a way to go...

[0] https://github.com/Alephbet/alephbet

[1] https://github.com/Alephbet/gimel

[Disclaimer: I previously worked for Optimizely as predictive analytics PM - but no longer work there, and don't speak for the company.]

Optimizely has a bandit based 'traffic auto-allocation' feature in production on select enterprise plans [1]; bandits are excellent in a wide range of situations, and have many advantages, but like anything, have design parameters and there are some caveats you have to be aware of to make sure you are using them effectively.

On Frequentist and Bayesian: Optimizely's stats engine combines elements of both Frequentist and Bayesian statistics. They have a blog that tries to touch on this issue [2] But this is subtle stuff - and there are a lot of trade-offs, and different perspectives; look at the Bayesian/frequentist debate which has been going on for decades among statisticians.

But, FWIW, I definitely saw Optimizely as an organisation make a big investment to produce a stats engine which had the right trade-offs for how their customers were trying to test; and I think the end result was way more suitable than 'traditional' statistics were.

[1] https://help.optimizely.com/hc/en-us/articles/200040115-Traf... "Traffic Auto-allocation automatically adjusts your traffic allocation over time to maximize the number of conversions for your primary goal. [...] To learn more about how algorithms like this work, you might want to read about a popular statistics problem called the “multi-armed bandit.”"

[2] https://blog.optimizely.com/2015/03/04/bayesian-vs-frequenti... "Yet as we developed a statistical model that would more accurately match how Optimizely’s customers use their experiment results to make decisions (Stats Engine), it became clear that the best solution would need to blend elements of both Frequentist and Bayesian methods to deliver both the reliability of Frequentist statistics and the speed and agility of Bayesian ones."

Hi Fergal!

I didn't realize that the auto-allocation ever shipped, but I'm glad it finally did. Hopefully there was work done to empirically show that they solved a lot of the issues around time to convert and other messy parts of the data that killed earlier efforts, but I think everyone who knew about those was gone before you joined :)

There are very subtle issues with both frequentist and bayesian stats, which makes combining them sounds insane to me.

What are you up to these days?

There's a reason you'll only see their number say >99%

You're wrong. What you are thinking about is Optimizely's "Chance to Beat Baseline" number. That's different from the statistical significance, which is a setting you can change on the Settings page.

Being smug and condescending really backfires when you don't know what you're talking about.

Bro... http://cl.ly/1b0B3Y3o1w09

> Being smug and condescending really backfires when you don't know what you're talking about.

How's that working out for you?

Running their numbers:

Assuming the usual sqrt(n+1) error on a count, as n is low. Combining the uncertainties from 83/66 [0] gives 1.258 +- 0.231 (1sd). It's only just likely that it is an actual improvement, but there is about a 30% chance that the orignal was better.

[0] (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2)^0.5x83.0/66

EDIT

If we also include the errors on the total counts and use the fraction 83.0/66/6362x6392, then the error is:

(((1+66^0.5)/66)^2+((1+83^0.5)/83)^2+((1+6362^0.5)/6362)^2+((1+6392^0.5)/6392)^2)^0.5x83.0/66/6362x6392

which shows an improvement of 1.264+-0.234. Stastically nout.

The fact that on the summary page there isn't a number showing the significance is very worrying. As the whole reason for it is to make a binary choice, there could be a 2 line chuck of js that just says "B might be better (60% probablity)".

I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia:

    using Distributions

    b_old = Beta(66+1, 6392-66+1)
    b_yel = Beta(83+1, 6362-83+1)

    N = 1000000
    # Sample from both distributions, count the fraction of samples that are better
    sum(rand(b_old, N) .> rand(b_yel, N)) / N

This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers to a simple question.

Why is that answer so different? It's saying there's a 90%+ chance that the new one is better. That's also his answer.

Thanks for this analysis.

I did an A/B test on an older framework which didn't automate statistical significance at all, but the website was getting more than 2000-3000 orders per day, so after a single week we had enough data to determine that sales had increased by 36% (reduced the page's load time by almost half, changed the checkout to use Ajax, and a few other small changes.) without the need to quantify things. In fact, at the time, I didn't even know what "statistical significance" was... not that I know too much more about statistics now than I did then.

Anyway, in theory all of the exact p values, etc. matter, but in practice, the bottom line is all that matters, because the p values can change in a moment based on something I haven't factored in. That's where, at least for the time being, intuition still plays a great part in being actually correct, which is why we still have people with repeated successes.

Good point, there are always unmodeled factors and prior information, so the % is to be interpreted in context.

I get about the same result using my favorite online Bayesian split test calculator: http://www.peakconversion.com/2012/02/ab-split-test-graphica...

" If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc."

I'd serve them at the same time, randomize which clients see which one.

And it turns out you randomly assigned B to a poorer user more often than not, and this make B worse...

That's why you do it properly randomized. Statistics tell you how much of a risk of your scenario (and similar) you are running at most.

Ofcourse, that's what A/B testing is about. But this only works if you have lots of traffic. The smaller the amount of traffic, the bigger the influence of offsite factors. Also, the results become more difficult to explain: https://news.ycombinator.com/item?id=11438098

seasonality doesn't really come into it with A/B split testing - if it changes for one group it changes for both.

Let's assume your site is now working with option A. Conversion rate is 5.7%, measured over a month.

You're now running an A/B for 1 week with option A and another version, option B. You get a conversion rate of 6.5% for option A and 5.9% for option B. Normally, you'd say that 6.5% is better than 5.9%. But how sure can you be if you don't control the other factors and both are performing better than before? How many visitors do you need to offset the influence of offsite factors?

Set your site up to be able to handle option A and option B at the same time, randomly assigning visitors to each. Now you don't need to control for anything, because you can just compare the A-visitors to the B-visitors.

In your case, option A started doing a lot better when you started running the test, which is moderately surprising, so maybe you messed something up in user assignment, tracking, or something. But if your framework is good, then it sounds like an external change. You still want to look at how A's rate and B's rate compare for the time period when you were randomizing, so 6.5% vs 5.9%, with the earlier 5.7% being more or less irrelevant.

You do need to do significance testing to see how likely you would be to get this wide a difference by chance. The easiest way to do that is to enter your #samples and #conversions into a significance calculator [1], and the p-value tells you how likely this is (assuming you didn't have any prior reason to expect one side of the experiment to be any better).

[1] Like this one https://vwo.com/ab-split-test-significance-calculator/ , but multiply the p-values it gives you by 2 to account for them incorrectly using a one-tailed test.

To ignore seasonality requires assuming it has roughly the same effect on both groups. If they move in different directions or by very different amounts, you can't actually ignore it.

Simple example: you're comparing your standard site's theme with a new pumpkin orange theme. In October, the pumpkin orange theme might go great, whereas in December, it might perform worse. There's clearly a seasonality interaction you need to account for.

What you're thinking of is more like testing e-commerce button colors right after Christmas. After Christmas, it's likely that all your groups perform worse, but equally so.

I think the goal with this new mechanism is having the script "automatically" promote your Christmas-themed creatives in December, and go back to your normal creatives in January.

It seems like you could "bucket" results by week, to get some kind of date factor considered. Of course it's also possible your Christmas creatives are simply garbage and will never get promoted, even when they are in-season. ;)

Reply on news.ycombinator.com