Most Winning A/B Test Results Are Illusory [pdf]
qubit.com
qubit.com
Thompson sampling avoids the problems of multiple testing, power, early stopping and so on by starting with a proper Bayesian approach. The idea is that the question we want to answer is more "Which alternative is nearly as good as the best with pretty high probability?". This is very different from the question being answered by a classical test of significance. Moreover, it would be good if we could answer the question partially by decreasing the number of times we sample options that are clearly worse than the best. What we want to solve is the multi-armed bandit problem, not the retrospective analysis of experimental results problem.
The really good news is that Thompson sampling is both much simpler than hypothesis testing can be done in far more complex situations. It is known to be an asymptotically optimal solution to the multi-armed bandit and often takes only a few lines of very simple code to implement.
See http://tdunning.blogspot.com/2012/02/bayesian-bandits.html for an essay and see https://github.com/tdunning/bandit-ranking for an example applied to ranking.
Thompson Sampling (at least the standard approach) assumes that conversion rates do not change. In reality they vary significantly over a week, and this fundamentally breaks bandit algorithms.
https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm...
Furthermore, you do not need to use Thompson Sampling to have a proper Bayesian approach. At VWO we also use a proper Bayesian approach, but we use A/B testing in to avoid the various pitfalls that Thompson Sampling has. Google Optimize uses an approach very similar to ours, (although it may be flawed [1]) and so does A/B Tasty (probably not flawed).
https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic...
Note: I'm the Director of Data Science at VWO. Obviously I'm biased, etc. However my post critiquing bandits was published before I took on this role. It was a followup to a previous post of mine which led people to accidentally misuse bandits: https://www.chrisstucchio.com/blog/2012/bandit_algorithms_vs...
[1] The head of data science at A/B Tasty suggests Google Optimize counts sessions rather than visitors, which would break the IID assumption. https://www.abtasty.com/uk/blog/data-scientist-hubert-google...
> Depending on what your website is selling, people will have a different propensity to purchase on Saturday than they have on Tuesday.
Affects multi-armed bandit and fixed tests. If you do fixed A/B test on Tuesday, your results will also be wrong. Either way, you have to decide on what kind of seasonality your data has, and don't make any adjustments until the period is complete.
If anything, multi-armed bandit shines because it can adapt to trends you don't anticipate.
> Delayed response is a big problem when A/B testing the response to an email campaign.
Affects multi-armed bandit and fixed tests. If you include immature data in your p-test, your results will be wrong. Either way, you have to decide how long it takes to declare an individual success or failure.
> You don't get samples for free by counting visits instead of users
Affects multi-armed bandit and fixed tests. Focusing on relevant data increases the power of your experiment.
---
For every single problem, the author admits "A/B tests have the same problem", and then somehow concludes that multi-bandit tests are harder because of these design decisions, despite the fact they affect any experiment process.
It can, but the time it takes is exp(# of samples already passed).
You can improve this by using a non-stationary Bayesian model (i.e. one that assumes conversion rates change over time) but this usually involves solving PDEs or something equally difficult.
For every single problem, the author admits "A/B tests have the same problem", and then somehow concludes that multi-bandit tests are harder because of these design decisions, despite the fact they affect any experiment process.
The point the author (me) is trying to make is not that bandits are fundamentally flawed. The point is that for A/B tests, all these problems have simple fixes: make sure to run the A/B test for long enough.
For bandits, the fixes are not nearly as simple. It usually involves non-simple math, or at the very least non-intuitive things (for instance not actually running a bandit until 1 week has passed).
At VWO we realized that most of our customers are not sophisticated enough to get all this stuff right, which is why we didn't switch to bandits.
Multi-bandit has the same fix: make sure the test has run for long enough before adjusting sampling proportions.
You seem to be proposing to run A/B with a 50/50 split for a full week, then when B does a lot better shift to 10/90 in favor of B and maybe a few weeks later shift to 1/99.
What practical benefit do you see to this approach? From my perspective this just slows down the experimental process and keeps losing variations (and associated code complexity) around for a lot longer.
> What practical benefit do you see to this approach?
Statistically rigorous results, with minimal regret.
In your example, you reach the end of the week, and your 50/50 split has one-sided p=0.10, double the usual p<0.05 criteria. What do you do?
(a) Call it in favor of B, despite being uncertain about the outcome. (b) Keep running the test. This compromises the statistical rigor of your test. (c) Keep running the test, but use sequential hypothesis testing, e.g. http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigo... This significantly increases the time to reach a conclusion, and costs you conversions in the meantime.
(a) and (b) are the most popular choices, despite them being statistically unjustifiable. http://www.evanmiller.org/how-not-to-run-an-ab-test.html
---
The essential difference when choosing the approach is that 50/50 split optimizes for shortest time to conclusion, and multi-bandit optimizes for fewest failures.
In web A/B testing, the latter is usually the most applicable, and for that, you cannot beat Thompson sampling on the average, no matter how diabolically clever your scheme. https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of...
There are times when the former is more important, e.g. marketing wants to know how to brand a product that is being released next month. These are the clinical-like experiments that frequentist approaches were formulated for.
The results are only statistically rigorous provided your bandit obeys relatively strong assumptions.
As another example, suppose you ran a 2-week test. Suppose that from week 1 to week 2, both conversion rates changed, but the delta between them remained roughly the same. A 50/50 A/B split doesn't mind this, and in fact still returns the right answer. Bandits do mind.
I don't do p-values. I do Bayesian testing, same as you. I just recognize that in the real world, weaker assumptions are more robust to experimenter or model error, both of which are generally the dominant error mode.
In web A/B testing, the latter is usually the most applicable, and for that, you cannot beat Thompson sampling on the average, no matter how clever yur scheme.
This is simply not true. The Gittins Index beats Thompson sampling, subject again to the same strong assumptions.
Look, I know the theoretical advantages of bandits and I advocate their use under some limited circumstances. I just find the stronger assumptions they require (or alternately the much heavier math requirements) mean they aren't a great replacement for A/B tests which are much simpler and easier to get right.
Yes, if you assume stability and things vary, you will not have good results. That seems like any statistics.
https://www.chrisstucchio.com/blog/2013/time_varying_convers...
However, with an A/B test you don't need to change the math or eliminate the stability assumption from it. You just need to choose a good test duration.
As I pointed out to pauldraper in the other thread, when you start fixing bandits by only changing the split every season, suddenly bandits start to look a lot like A/B testing.
Thompson sampling (and Bayesian Bandits in general) can be applied with a model for estimating conversion that is more complex than P(conversion|A). It can include parameters for time of day, day of week and even be non-parametric.
If you do this, you the standard Thompson sampling framework with nochanges* whatsoever will still kill losers quickly (if the loss is big enough that seasonality cannot save it) and will also wait until it has enough data seasonally to make more refined decisions. This is very different from simply waiting for an even season point to make decisions.
You do need more data to understand a more complex world, but having an optimal inference engine to help is better than having a severely sub-optimal engine.
But the point I'm making is different. This is a lot of stuff to get right and most people aren't that sophisticated. Getting A/B tests right is a lot easier, mainly because they are significantly more robust to model error.
It's often the case that the decision maker has already decided to move ahead with option A, but performs a minimal "fake" A/B test to put in their report as a way to justify their choice. I've seen A/B tests deployed at 10am in the morning, and taken down at 1pm with less than a dozen data points collected. The A/B test "owner" is happy to see that option A resulted in 7 conversions, with option B only having 5. Not statistically significant whatsoever, but hey let's waste developers' time and energy for two days implementing an A/B test in order to help someone else try to nab their quarterly marketing bonus.
Move your decision process to multi-armed bandit and you never have to decide when to end an A/B test -- math does it for you, in a provably optimal way.
IDK about your intuition, but for most other people, it gets in the way of statistics.
The "loss function" is just as easy to calculate for A/B tests as for multi-armed bandit. The value of user doing A is $X, the value of B is $Y, and the value of C is $Z.
Naive Thompson sampling (like the code you linked to) will result in problems equally disastrous to those I wrote about in the Qubit whitepaper. Other comments have highlighted a key problem with simple bandit algorithms - reward distributions which change over time will render their results worthless. You can model these dynamics but not in 'a few lines of very simple code'. It is verging on the irresponsible to suggest otherwise.
I personally favour a bayesian state-space model to elegantly take care of these things - but that's outside the remit of the whitepaper and outside the skill set of most non-statisticians. Frequentist testing, when done properly, is simple to implement and has statistical guarantees that are very attractive in practice.
I wrote a blog post focusing on the #2 problem mentioned: multiple testing. I simulated the typical mitigating approaches, and comparing them against Thompson sampling (code in GitHub). https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of...
I'm not a Bayesian fanatic, but given how perfectly A/B test optimization fits in the Bayesian approach, it's a shame it's not yet the de facto standard.
I think the primary reasons are (1) it's not as intuitive (especially for the uninitiated), (2) it's harder to implement an automated feedback mechanism, (3) FUD. E.g. https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm... lists devastating complications with correct multi-armed bandit tests, and then in fine print admits that traditional tests have all the same complications.
But it's easier to try to black box it and say this number cruncher tells me if these two things are really different. Of course, if you don't understand what's going on, you start p-hacking.
Anyway, you're I think you're right.
The goal is to make practical business decisions with reasonably high likelihood of being right and, if wrong, having only limited impact.
The output of a well done Bayesian analysis can be very easy for business people to understand. Statements like (you have a 60% chance of making just the right decision and a 35% chance of making a decision that is within 3% of right) are easy for most business stake-holders to understand because that is close to how they frame their own decision making process.
Laboring to reject a null hypothesis is an unnatural act for most people.
Firmly believe that there's way more to be gained by more people understanding how to use A/B testing than more complex solutions.
I didn't persevere because the R language rubs me the wrong way and I'd be comfier with Python or Haskell or something. http://camdavidsonpilon.github.io/Probabilistic-Programming-... looks promising in that vein, but I haven't read it.
That being said, building a "proper" a/b harness is really hard and will be a constant source of bugs / FUD around decision-making (don't believe me? try running an a/a experiment and see how many false positives you get). I've personally built a dead-simple bandit system when starting greenfield and would recommend the same to anyone else.
Optimizely takes an interesting approach: they apply repeat testing methods, segmenting the tests by user views of the results. Like 30x more complicated than multi-bandit, but they don't need a feedback mechanism.
By the way, I doubt you remember me, but thank you for inviting me on a tour of Veoh ten years ago when I was a young college sophomore. I enjoyed the opportunity as well as our brief chat about Bayesianism.
On the other hand, one reason is that I invited a lot of students to come visit (and a number to intern with us).
The right question to ask is: What is the difference between A and B, and what is our uncertainty on that estimate?
The wrong question to ask is: Is A different/better than B, given some confidence threshold?
The reason this is the wrong question is that it's unnecessarily binary. It is a non-linear transformation of information that undervalues confidence away from the arbitrary threshold and overvalues confidence right at the arbitrary threshold.
A test with only 10 or 100 samples still gives you information. It gives you weak information, sure, but information nonetheless. If you approach the problem from a continuous perspective (asking how big the difference is), you can straightforwardly use the information. But if you approach the problem from a binary hypothesis-testing perspective (asking is there a difference), you'll be throwing away lots of weak information when it could be proving real (yet uncertain) value.
Once you switch away from the binary hypothesis-testing framework, you no longer have to worry about silly issues like stopping too early or false positives or false negatives. You simply have a distribution of probabilities over possible effect sizes.
Before you can quantify a difference, you have to determine whether one exists in the first place. That is the purpose of binary testing; without it, you're just looking at noise without any means to decide what is real and what is not.
As a corollary, if you can meaningfully quantify the difference between A and B, then you should have no trouble establishing that they are different. Obviously business decisions are not generally going to uphold the rigor of good science, but what is the purpose of quantifying things when you're as likely to be wrong as your are right?
The superior question is which is probably better, and by how much. If all you know is that A is 75% likely to be better than B, then go with A. It's useful information, even if it doesn't cross a an arbitrarily preset threshold of 95% or whatever you use.
You don't need to wait for certainty to act. In fact, all actions are taken under uncertainty. So it feels incredibly artificial and counterproductive to frame these questions in such a binary, nonlinear way. It's clinging to certainty when certainty does not exist.
The best question is what's the expected reward from choosing A.
Consider the scenario of 10% chance B > A, 90% chance they are equal. To me that sounds like a winning bet.
In contrast, 70% chance B is a lot better than A, 30% chance B is a lot worse than A sounds like a scary bet. I'll wait and gather more info.
Proper decision theory is based on a loss function which incorporates the magnitude of gains/losses rather than just their existence.
Disclaimer: I wrote that article (a long time ago...)
The statistical rigour of all the hypothesis tests and Bayesian methods and so on is laudable, but fundamentally any result and any conclusion you draw from those tests can only ever be as good as the underlying assumptions from which the result was derived. For example, if you choose to perform any statistical test that designates one of A and B the default case and tests for evidence in favour of the other, you have made a determination that the situation is not symmetric. You can choose a level of power or a significance level you want to achieve, but the 5% convention is again arbitrary. The same goes for any prior you choose in a Bayesian method.
Fundamentally, if you are truly starting from a neutral situation where you just want to know which of A and B is better, and you truly have no other information to go on other than how they have performed since the start of your measurements with all else being equal, then it remains the case that you still have nothing else to go on no matter what statistical calculations you perform.
That's a perfectly fine question so long as there are three acceptable answers: Yes, No and Unclear.
It feels so pointless to bin probabilities into yes, no, and unclear. We're throwing away information with value when we compress the problem this way. And I think it leads to more misinterpretations.
If your market consists of 1000 people.
This video is helpful: https://www.youtube.com/watch?v=Dy_LRK2Pkig
I still don't know a great way to describe it in the 6 or 7 seconds you have before the potential customer's attention starts to flag.
I don't think there is a 6-7 second way to describe it. I mean, imagine someone that knows nothing of frequentist statistics, could you explain the meaning/significance of a confidence interval to them? It's a pretty high bar to clear, I've found for CI's it's typically 1-5 minutes depending on background.
Instead, they should be focusing on the quality of the tests they run. Quality hypotheses, preferably backed by data-driven insights into behavior that test very clear, if not dramatic, changes to the user flow will be what leads to bottom-line improvements, not increasing the mathematical merit and rigor of poorly-conceived tests. Of course, doing both is ideal but I put more fault on the testing software than the companies using it.
Keep track of historical conversion rate and adjust/account for noise in your historical conversion rate. Conceive a testing program that focuses on the quality testing and you'll likely see an upward trend in that historical conversion rate.
You can see a demonstration of this in practice: http://www.gigamonkeys.com/interruptus/ (sit back and watch your false positive rate, aka "bogus a/b testing results", grow).
Also, the more tests you do the lower the p-value should be.
If you stop when you reach your p-value then you are misinterpreting the results if you claim the results mean anything.
Now consider the following two scenarios:
(1) After giving drug B to 100 patients, all 100 patients are dead. Do you continue the trial, giving the (apparently) deadly drug B to 900 more patients?
(2) After giving drug B to 100 patients, all 100 patients are totally cured (vs A curing 3 in 100). Do you continue the trial, withholding the (apparent) cure for cancer from 900 more patients?
In either case, since you have a strong effect, it seems to me there is logical justification to end the trial early.
Obviously the stakes are higher in clinical trials than website design, but in both cases, data acquisition has costs and intermediate results may inform changes to your experiment design.
I honestly cannot see how anyone could blankly assert that stopping a test is always wrong. There are certainly circumstances where you do want to stop early. You just have to make sure you aren't misinterpreting a statistic when you do so.
One of the simplest ways to end a trial early is through curtailment--you stop the trial when additional data won't change its outcome. Imagine you have a big box of 100 items, which can either be Item A or Item B. You want to test whether the box contains an equal proportion of As and Bs, so you pull an item from the box, unwrap it, and record what is inside. Naively, one might think that it is necessary to unwrap all 100 items, but you can actually stop after you find 58 of the same type because the 58:42 split--and all more extreme imbalances--allows you to reject the 50:50 hypothesis.
Curtailment is "exact" and fairly easy to compute if you have a finite sample size, each of which contributes a bounded amount to your result. This would certainly happen in your extreme examples. There are also more complicated approaches that allow you to stop even earlier if either a) you're willing to sometimes make a different decision than if you ran to completion and/or b) you're willing to stop earlier on average, even if it means running for longer in some cases.
Take your example. 100 patients, all dead or 100 patients all alive -- you have demonstrated an infinite effect (edit: ok no need to be hyperbolic, 99+% effect), and probably have covered statistical rigor. If your drug is that effective you are probably criminally liable a lot sooner than 100 dead patients. Unfortunately actual medical (and A/B) studies do not mimic make believe scenarios.
If you "peek and halt" without doing the math. You might as well have a random good result in the first 10 and say "look! Positive results!". You recognize that is ridiculous. So when is peeking and stopping not ridiculous?
A: when the statistical power is sufficient for the observed effect. In the examples -- 1600 or 6000 for a 10% or 5% effect, respectively. And much less for a 20% or 40% effect! -- but you don't know the number required unless you do the math.
You can even get an economic estimate of the expected amount of value you are leaving on the table by stopping at any point.
Above there are a number of things to notice:
1) The phasing approximates Thompson sampling to a degree, in that large late-phase trials MUST follow smaller early phase trials. Nobody is going to waste patients on SuperMab (look it up).
2) The endpoints are hard, fast, and pre-specified:
IFF we have N adverse events in M patients, we shut down the trial for toxicity.
IFF we have X or more complete responses in Y patients, we shut down the trial because it would be unethical to deprive the control arm.
IFF we have Z or fewer responses in the treatment arm, given our ultimate accrual goal (total sample size), it will be impossible to conclude (using the test we have selected and preregistered) that the new drug isn't WORSE than the standard, so we'll shut it down for futility. Those patients will be better served by another trial.
You are massively oversimplifying a well-understood problem. Decision theory is a thing, and it's been a thing for 100 years. Instead of lighting your strawman on fire, how about reframing it?
Stopping isn't "always" wrong, but stopping because you've managed to hit some extremal value is pretty much always biased. The "Winner's curse", regression to the mean, all of these things happen because people forget about sampling variability. It's also why point estimates (even test statistics) rather than posterior distributions are misleading. If you're going to stop at an uncertain time or for unspecified reasons, you need to include the "slop" in your estimates.
"We estimate that the new page is 2x (95% CI, 1.0001x-10x) more likely to result in a conversion"... hey, you stopped early and at least you're being honest about it... but if we leave out the uncertainty then it's just misleading.
All of the above is taken into account when designing trials because not only do we not like killing people, we don't like going to jail for stupid avoidable mistakes.
It isn't bullshit - at least, not for that reason. It was a thought experiment, using an extreme example to test your assertions.
In the future, I would appreciate it if you steelmanned my comments or asked for clarification instead of insulting me. It hurt my feelings. I wish I had written a better comment that hadn't incited such a reaction from you. Best wishes.
I also have noprocrast set in my profile so I couldn't go back and edit it (something I thought about doing). I probably would have toned it down if I hadn't requested that Hacker News kick me off after 15 minutes.
Your line of discussion is productive. It's just important that people understand the difference between degrees of belief and degrees of evidence from a specific study and never confuse the two. Trouble is, lots of folks confuse them, and lots of other folks prey on that confusion.
Anyways, sorry for being a jerk.
But it applies at all scales of effect. Stop when you have a big enough effect or have high confidence that you won't care.
a) when the challenger strategy is much worse than the best alternative with reasonable probability,
or
b) when it is clear that some strategy is very close to the best with high probability (remember, you may still not know how good the best is)
Note that waiting for significance kind of handles the first case, but in the case of ties, it completely fails on the second case. That is, when there are nearly tied options, you will wait nearly forever. It is better to pick one of the tied options and say that it is very likely as good as the best.
(1) would absolutely not be continued, obviously.
(2) would be continued to completion.
In science one is maximally conservative with positive results. Negative results can be accepted rather quickly but positive results need to be maximally investigated. False positives are unacceptable, false negatives are often less damaging.
I'd question how scenario #1 made it through animal trials preceding experiments in humans.
This isn't totally uncontroversial (e.g., https://www.ncbi.nlm.nih.gov/pubmed/18226746 ) but it's not obvious to me how to balance statistical and ethical concerns.
As for the animal studies, it sometimes happens. Someone up-thread mentioned SuperMAB, a potential drug which seemed safe in monkeys. However, when first tested in humans (using a dose 500x lower than the one administered to the monkeys), it generated a cytokine storm that put all six test subjects in the ICU. Here's a fairly decent summary: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2964774/
Source: I have a degree in analytics.
If you're planning to stop as soon as you find a positive result you'll need to modify the tests to ensure that the total chance that any test results in a positive is low enough. In general you'll need to keep increasing the significance level as you do more events (if you only plan to test a finite number of times you can keep the significance fixed, but I think this will lead to more false negatives than an 'adaptive' significance).
To illustrate, if you have 6000 events and check for 99% significance after every one, then you'd expect about 60 false positives on average. Of course these false positives aren't distributed uniformly, so it's not like you'll always find 60 false positives, however it's not like you'll only find 0 or 6000 either, meaning that (significantly) more than 1% of the time you'll have at least 1 false positive.
Of course there's the generic Bonferroni correction where you just divide the significance threshold by the number of tests, but you'll end up with a lot more false negatives (where you just miss the result entirely), and if you want to keep running the test for arbitrary lengths of time it becomes even more difficult (you'll need to keep lowering the threshold). Then again this does give strong guarantees and might work well enough if you don't check the results often.
Basically to get the best results you'll need to balance the false negative rate against the false positive rate and the expected length of the test, which is a rather complicated trade off. But I expect someone will have done at least some of the calculations for simple A/B testing.
People got tired of trialists playing games with statistics and patients dying or ineffective drugs making it to market. Now if you want to run a trial as evidence for approval, you need to specify an endpoint, how it will be tested, when, with what alpha (false positive) threshold, and what's the minimum effect size required for this.
If you are doing interim analysis, futility, or non-inferiority, you have to write that into the design, too.
People can jerk around with subgroup analyses in publications but the FDA won't accept that sort of horse shit for actual approval. And thank heavens for that.
He ran some simulated A/B tests using my Bayesian A/B testing technique (which is now powering VWO). He showed that while peeking does ensure that loss < specified threshold, if you don't peek the loss will be even lower.
So although peeking is still valid, that validity does come at a cost.
Honestly, if the first 100 patients die into a 1,000-patient clinical trial, I have zero qualms about making the judgment to stop early, even if it wasn't written into the design. I'm not going to kill 900 people by religiously following bad statistical principles.
I think we should be open-minded and understand that sometimes peeking is ok and sometimes it isn't.
When the effect is large, you can end earlier. There's no reason to cling to a formula and procedure that requires a fixed number of samples when other methods exist that lack that drawback.
"It's totally reasonable to roll your own cryptography, as long as you're an expert in the field and do it correctly".
The rules of "Don't Stop And Peek" is general advice that is given out because the vast majority of people that conduct these kinds of trials are not statisticians, and will not do the math. Those that attempt the math will likely do the math wrong.
So, you're totally correct that if you know exactly what you're doing as a statistician and apply the math correctly, it can be possible to peek at results while still drawing a conclusion. This ignores the fact that, for most humans who are receiving the advice, this is terrible advice.
So, general rule: "Don't Stop and Peek". Advanced rule: "Unless you are a trained statistician, Don't Stop and Peek".
Similarly, every test you do has a small probability to give a false positive, the more test you do the bigger the total chance that you'll be jumping to conclusions.
Also, the size of the effect is irrelevant since that should already be accounted for by whatever test you do.
Anyway, I don't even think false positives are the right way to think about this. The framing should be continuous, not binary. The goal is to maximize success, not maximize the odds of picking the better of A or B.
A Bayesian analysis tells you Bayesian things: specifically, this is the most reasonable conclusion one can draw from this data, right now. A frequentist analysis also tells you things, but they are different things. Specifically, frequentists are concerned with...frequencies: if we were to run this procedure many times, here's what we can say about the possible outcomes.
There's this persistent meme that all Bayesian methods protect you from multiple comparisons/multiple looks problems. That's not really true. Bayesian methods don't offer any Type I/Type II error control--and why would they, when the notion of these errors doesn't make much sense in a Bayesian framework?
You can certainly use Bayesian methods to estimate a parameter's value. However this you cannot repeatedly test your estimates--Bayesian or otherwise--in a NHST-like framework and expect things to work out correctly.
"You can perform power calculations by using an online calculator or a statistical package like R. If time is short, a simple rule of thumb is to use 6000 conversion events in each group if you want to detect a 5% uplift. Use 1600 if you only want to detect uplifts of over 10%. These numbers will give you around 80% power to detect a true effect."
These numbers assume approximately 10% of changes give true uplift (aka actual improvement). If the incidence of improvement is rarer you would need more than 6000 events to identify a 5% increase.
(OT: that term sounds like cult language -- you have three chances to achieve true uplift)
FYI, the difference with Google Analytics and most other A/B platforms is that GA uses multi-armed bandit. It adjusts the proportions according to the results. This is nice because it effectively ends the test for you, and you don't have to stress about suboptimal success rates just for the purpose of experimentation. https://support.google.com/analytics/answer/2844870
Yes.
> The larger the relative difference, the fewer events needed.
Wrong. That's the common mistake that is being addressed.
The relative difference could be anything at any point in time. You always need a lots of events to be somewhat confident that there is really a difference.
Short version: If you want X% confidence on a test, there are maths formula to say that you need to monitor N events, at least. The formula doesn't depends on how large is the difference (which you don't know anyway).
> Perform a second ‘validation’ test repeating your original test to check that the effect is real
Isn't this just the same as taking a larger sample size?
The difference is time, so the populations involved are 'the people that used the site last month' and 'the people that used the site last week'. Usually your assumption is that these are comparable, but that's not necessarily true. Furthermore, the effect is only meaningful if that assumption is true, for most business cases (i.e. you want this effect to hold in the future).
In practice, a lot of effects do disappear when you repeat a test because there was some unaccounted for factor that varied between them. It's a good sanity check.
But you're right, much of the purpose is to discover mean regression. This is something that happens more often than you'd expect because you tend to be focusing large effects, many of which will simply be due to chance.
Assume your first samples show the effect and the later ones don't. The first test will show a strong correlation and the second not. The larger test will simply show a weaker correlation.
The actual effect depends on your data and how noisy it is, and if that correlates to external factors, etc.
You need to know the characteristics of your data then you can decide on your desired confidence and only then pick a test designed to give you that based on your data.
You'd potentially lose 5% CR if you ship the variant even when it doesn't show a detectable uplift. Why would you do that?