Hacker News Reader

Comment by rco8786 | Hacker News Reader

You really should question the statistical significance of those numbers.

Running their numbers:

Assuming the usual sqrt(n+1) error on a count, as n is low. Combining the uncertainties from 83/66 [0] gives 1.258 +- 0.231 (1sd). It's only just likely that it is an actual improvement, but there is about a 30% chance that the orignal was better.

[0] (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2)^0.5x83.0/66

EDIT

If we also include the errors on the total counts and use the fraction 83.0/66/6362x6392, then the error is:

(((1+66^0.5)/66)^2+((1+83^0.5)/83)^2+((1+6362^0.5)/6362)^2+((1+6392^0.5)/6392)^2)^0.5x83.0/66/6362x6392

which shows an improvement of 1.264+-0.234. Stastically nout.

I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia:

    using Distributions

    b_old = Beta(66+1, 6392-66+1)
    b_yel = Beta(83+1, 6362-83+1)

    N = 1000000
    # Sample from both distributions, count the fraction of samples that are better
    sum(rand(b_old, N) .> rand(b_yel, N)) / N

This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers to a simple question.

Why is that answer so different? It's saying there's a 90%+ chance that the new one is better. That's also his answer.

Thanks for this analysis.

I did an A/B test on an older framework which didn't automate statistical significance at all, but the website was getting more than 2000-3000 orders per day, so after a single week we had enough data to determine that sales had increased by 36% (reduced the page's load time by almost half, changed the checkout to use Ajax, and a few other small changes.) without the need to quantify things. In fact, at the time, I didn't even know what "statistical significance" was... not that I know too much more about statistics now than I did then.

Anyway, in theory all of the exact p values, etc. matter, but in practice, the bottom line is all that matters, because the p values can change in a moment based on something I haven't factored in. That's where, at least for the time being, intuition still plays a great part in being actually correct, which is why we still have people with repeated successes.

Good point, there are always unmodeled factors and prior information, so the % is to be interpreted in context.

I get about the same result using my favorite online Bayesian split test calculator: http://www.peakconversion.com/2012/02/ab-split-test-graphica...

Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good.

So: probably better, but should have run it longer.

The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand...

I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained the same for the entire test, rather than there being a specific period where "yellow" excelled.

> I wanted Optimizely to say it was 100% significant

That's not how statistical significance works...

You should contact Optimizely and let them know right now.

https://www.optimizely.com/contact/

They'll need to reeducate their statisticians right away!

Optimizely did an A/B test that showed that customers respond better to rounded-up numbers. ;-) the real world is messy.

Years ago my team's statistician did a competitive review of various AB test apps, and reported various ways in which the UIs make statistically invalid statements to the user.

They probably do, but why would they want to do so, when doing it would make their results appear less conclusive?

The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about the size of the differences. It also can't be 100. The normal distribution has an infinite range, so a finite limit can never capture 100% of it.

Optimizely is in a rough spot. People don't like having to think through experimental design, and they are really, really bad at reasoning about p-values. To try to fix the people part, they came out with the sequential stopping rule stuff (their "stats engine), but they never really published much justifying it. The other alternative would be to move the experiments into a Bayesian framework, but that has a lot of it's own problems. When they acquired Synference, that was one of the likely directions to take (along with offering bandits), but that didn't work out and those guys have since left.

> but they never really published much justifying it

Not that I'm trying to defend Optimizely (I'm not a huge fan, but for other reasons...).

I can't vouch for the quality either, but they did publish something about it[0] - that at least looks quite scientific. Happy to read any critique of course.

[0] http://pages.optimizely.com/rs/optimizely/images/stats_engin...

Latex is a wonderful way to make a marketing paper look like a scientific one. It doesn't accurately describe the method, but that isn't really its purpose. It's a more technical description of the blog post, meant for people using the product to understand some of the tradeoffs and get more accurate results.

They are still having people make very fundamentally flawed assumptions about the data, which results in incorrect conclusions, and they are still not presenting the results in a way that people correctly interpret them. That being said, those are really hard to solve, and models that would try to correct for them would likely require a lot more data and be overly conservative for more people.

What are your reasons for disliking Optimizely?

It may be marketing but I was able to implement a sequential A/B test based on it. Admittedly, I did need to do some work beyond merely copy/pasting an algorithm, but all I really needed to do was read their paper and some citations. I do believe that this document does describe a viable frequentist test and my implementation of it worked pretty well.

Disclaimer: I do stats work at VWO, an Optimizely competitor.

(Also if you want to read our tech paper, here it is: https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic... This describes our Bayesian approach, which we believe to be less likely to be wrongly interpreted by non-statisticians.)

Hi, this is Leo, Optimizely's statistician. If you're looking for a more scientific paper, maybe take a look at this one we wrote recently: http://arxiv.org/abs/1512.04922

Should have everything you would ever want to know about the method.

I agree with you that the problem of inference and interpretation between A/B data, algorithms, and the people who make decisions from them is a hard one and worth working on.

That said, I do think the two sources of error our stats engine addresses - repeatedly checking results, and cherry picking from many metrics and variations - did make progress in having folks correctly interpret A/B Tests. This did result in more conservative results, but the benefit was that the variations that do become significant are more trustworthy. I think this was absolutely the right tradeoff to make for our customers, and trustworthyness is a pretty important aspiration for stats/ML/data science in general.

Of course I did write the thing, so I'm not very impartial.

I was tempted to make a snarky comment about using LaTeX, but I'm not sure it's entirely fair. It doesn't seem like just a bunch of MarketingSpeak wrapped in LaTeX to be honest.

My issue with Optimizely are mainly how they essentially ditched us as (paying) customers. We were admittedly small-fish, but we were paying and were willing to pay more, but they switched to Enterprise-vs-Free without any middle grounds. Enterprise was way too expensive for us. Free didn't include essential features, so we were just stuck.

I ended up writing an open-source javascript A/B test client[0] (and recently also an AWS-lambda backend[1]), but it still has a way to go...

[0] https://github.com/Alephbet/alephbet

[1] https://github.com/Alephbet/gimel

[Disclaimer: I previously worked for Optimizely as predictive analytics PM - but no longer work there, and don't speak for the company.]

Optimizely has a bandit based 'traffic auto-allocation' feature in production on select enterprise plans [1]; bandits are excellent in a wide range of situations, and have many advantages, but like anything, have design parameters and there are some caveats you have to be aware of to make sure you are using them effectively.

On Frequentist and Bayesian: Optimizely's stats engine combines elements of both Frequentist and Bayesian statistics. They have a blog that tries to touch on this issue [2] But this is subtle stuff - and there are a lot of trade-offs, and different perspectives; look at the Bayesian/frequentist debate which has been going on for decades among statisticians.

But, FWIW, I definitely saw Optimizely as an organisation make a big investment to produce a stats engine which had the right trade-offs for how their customers were trying to test; and I think the end result was way more suitable than 'traditional' statistics were.

[1] https://help.optimizely.com/hc/en-us/articles/200040115-Traf... "Traffic Auto-allocation automatically adjusts your traffic allocation over time to maximize the number of conversions for your primary goal. [...] To learn more about how algorithms like this work, you might want to read about a popular statistics problem called the “multi-armed bandit.”"

[2] https://blog.optimizely.com/2015/03/04/bayesian-vs-frequenti... "Yet as we developed a statistical model that would more accurately match how Optimizely’s customers use their experiment results to make decisions (Stats Engine), it became clear that the best solution would need to blend elements of both Frequentist and Bayesian methods to deliver both the reliability of Frequentist statistics and the speed and agility of Bayesian ones."

Hi Fergal!

I didn't realize that the auto-allocation ever shipped, but I'm glad it finally did. Hopefully there was work done to empirically show that they solved a lot of the issues around time to convert and other messy parts of the data that killed earlier efforts, but I think everyone who knew about those was gone before you joined :)

There are very subtle issues with both frequentist and bayesian stats, which makes combining them sounds insane to me.

What are you up to these days?

There's a reason you'll only see their number say >99%

You're wrong. What you are thinking about is Optimizely's "Chance to Beat Baseline" number. That's different from the statistical significance, which is a setting you can change on the Settings page.

Being smug and condescending really backfires when you don't know what you're talking about.

Bro... http://cl.ly/1b0B3Y3o1w09

> Being smug and condescending really backfires when you don't know what you're talking about.

How's that working out for you?

The fact that on the summary page there isn't a number showing the significance is very worrying. As the whole reason for it is to make a binary choice, there could be a 2 line chuck of js that just says "B might be better (60% probablity)".

Reply on news.ycombinator.com