You can see a demonstration of this in practice: http://www.gigamonkeys.com/interruptus/ (sit back and watch your false positive rate, aka "bogus a/b testing results", grow).
Also, the more tests you do the lower the p-value should be.
If you stop when you reach your p-value then you are misinterpreting the results if you claim the results mean anything.
Now consider the following two scenarios:
(1) After giving drug B to 100 patients, all 100 patients are dead. Do you continue the trial, giving the (apparently) deadly drug B to 900 more patients?
(2) After giving drug B to 100 patients, all 100 patients are totally cured (vs A curing 3 in 100). Do you continue the trial, withholding the (apparent) cure for cancer from 900 more patients?
In either case, since you have a strong effect, it seems to me there is logical justification to end the trial early.
Obviously the stakes are higher in clinical trials than website design, but in both cases, data acquisition has costs and intermediate results may inform changes to your experiment design.
I honestly cannot see how anyone could blankly assert that stopping a test is always wrong. There are certainly circumstances where you do want to stop early. You just have to make sure you aren't misinterpreting a statistic when you do so.
Above there are a number of things to notice:
1) The phasing approximates Thompson sampling to a degree, in that large late-phase trials MUST follow smaller early phase trials. Nobody is going to waste patients on SuperMab (look it up).
2) The endpoints are hard, fast, and pre-specified:
IFF we have N adverse events in M patients, we shut down the trial for toxicity.
IFF we have X or more complete responses in Y patients, we shut down the trial because it would be unethical to deprive the control arm.
IFF we have Z or fewer responses in the treatment arm, given our ultimate accrual goal (total sample size), it will be impossible to conclude (using the test we have selected and preregistered) that the new drug isn't WORSE than the standard, so we'll shut it down for futility. Those patients will be better served by another trial.
You are massively oversimplifying a well-understood problem. Decision theory is a thing, and it's been a thing for 100 years. Instead of lighting your strawman on fire, how about reframing it?
Stopping isn't "always" wrong, but stopping because you've managed to hit some extremal value is pretty much always biased. The "Winner's curse", regression to the mean, all of these things happen because people forget about sampling variability. It's also why point estimates (even test statistics) rather than posterior distributions are misleading. If you're going to stop at an uncertain time or for unspecified reasons, you need to include the "slop" in your estimates.
"We estimate that the new page is 2x (95% CI, 1.0001x-10x) more likely to result in a conversion"... hey, you stopped early and at least you're being honest about it... but if we leave out the uncertainty then it's just misleading.
All of the above is taken into account when designing trials because not only do we not like killing people, we don't like going to jail for stupid avoidable mistakes.
In the future, I would appreciate it if you steelmanned my comments or asked for clarification instead of insulting me. It hurt my feelings. I wish I had written a better comment that hadn't incited such a reaction from you. Best wishes.
But it applies at all scales of effect. Stop when you have a big enough effect or have high confidence that you won't care.
I also have noprocrast set in my profile so I couldn't go back and edit it (something I thought about doing). I probably would have toned it down if I hadn't requested that Hacker News kick me off after 15 minutes.
Your line of discussion is productive. It's just important that people understand the difference between degrees of belief and degrees of evidence from a specific study and never confuse the two. Trouble is, lots of folks confuse them, and lots of other folks prey on that confusion.
Anyways, sorry for being a jerk.
It isn't bullshit - at least, not for that reason. It was a thought experiment, using an extreme example to test your assertions.
Take your example. 100 patients, all dead or 100 patients all alive -- you have demonstrated an infinite effect (edit: ok no need to be hyperbolic, 99+% effect), and probably have covered statistical rigor. If your drug is that effective you are probably criminally liable a lot sooner than 100 dead patients. Unfortunately actual medical (and A/B) studies do not mimic make believe scenarios.
If you "peek and halt" without doing the math. You might as well have a random good result in the first 10 and say "look! Positive results!". You recognize that is ridiculous. So when is peeking and stopping not ridiculous?
A: when the statistical power is sufficient for the observed effect. In the examples -- 1600 or 6000 for a 10% or 5% effect, respectively. And much less for a 20% or 40% effect! -- but you don't know the number required unless you do the math.
You can even get an economic estimate of the expected amount of value you are leaving on the table by stopping at any point.
(1) would absolutely not be continued, obviously.
(2) would be continued to completion.
In science one is maximally conservative with positive results. Negative results can be accepted rather quickly but positive results need to be maximally investigated. False positives are unacceptable, false negatives are often less damaging.
I'd question how scenario #1 made it through animal trials preceding experiments in humans.
This isn't totally uncontroversial (e.g., https://www.ncbi.nlm.nih.gov/pubmed/18226746 ) but it's not obvious to me how to balance statistical and ethical concerns.
As for the animal studies, it sometimes happens. Someone up-thread mentioned SuperMAB, a potential drug which seemed safe in monkeys. However, when first tested in humans (using a dose 500x lower than the one administered to the monkeys), it generated a cytokine storm that put all six test subjects in the ICU. Here's a fairly decent summary: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2964774/
a) when the challenger strategy is much worse than the best alternative with reasonable probability,
or
b) when it is clear that some strategy is very close to the best with high probability (remember, you may still not know how good the best is)
Note that waiting for significance kind of handles the first case, but in the case of ties, it completely fails on the second case. That is, when there are nearly tied options, you will wait nearly forever. It is better to pick one of the tied options and say that it is very likely as good as the best.
One of the simplest ways to end a trial early is through curtailment--you stop the trial when additional data won't change its outcome. Imagine you have a big box of 100 items, which can either be Item A or Item B. You want to test whether the box contains an equal proportion of As and Bs, so you pull an item from the box, unwrap it, and record what is inside. Naively, one might think that it is necessary to unwrap all 100 items, but you can actually stop after you find 58 of the same type because the 58:42 split--and all more extreme imbalances--allows you to reject the 50:50 hypothesis.
Curtailment is "exact" and fairly easy to compute if you have a finite sample size, each of which contributes a bounded amount to your result. This would certainly happen in your extreme examples. There are also more complicated approaches that allow you to stop even earlier if either a) you're willing to sometimes make a different decision than if you ran to completion and/or b) you're willing to stop earlier on average, even if it means running for longer in some cases.
Source: I have a degree in analytics.
If you're planning to stop as soon as you find a positive result you'll need to modify the tests to ensure that the total chance that any test results in a positive is low enough. In general you'll need to keep increasing the significance level as you do more events (if you only plan to test a finite number of times you can keep the significance fixed, but I think this will lead to more false negatives than an 'adaptive' significance).
To illustrate, if you have 6000 events and check for 99% significance after every one, then you'd expect about 60 false positives on average. Of course these false positives aren't distributed uniformly, so it's not like you'll always find 60 false positives, however it's not like you'll only find 0 or 6000 either, meaning that (significantly) more than 1% of the time you'll have at least 1 false positive.
People got tired of trialists playing games with statistics and patients dying or ineffective drugs making it to market. Now if you want to run a trial as evidence for approval, you need to specify an endpoint, how it will be tested, when, with what alpha (false positive) threshold, and what's the minimum effect size required for this.
If you are doing interim analysis, futility, or non-inferiority, you have to write that into the design, too.
People can jerk around with subgroup analyses in publications but the FDA won't accept that sort of horse shit for actual approval. And thank heavens for that.
Honestly, if the first 100 patients die into a 1,000-patient clinical trial, I have zero qualms about making the judgment to stop early, even if it wasn't written into the design. I'm not going to kill 900 people by religiously following bad statistical principles.
I think we should be open-minded and understand that sometimes peeking is ok and sometimes it isn't.
When the effect is large, you can end earlier. There's no reason to cling to a formula and procedure that requires a fixed number of samples when other methods exist that lack that drawback.
Similarly, every test you do has a small probability to give a false positive, the more test you do the bigger the total chance that you'll be jumping to conclusions.
Also, the size of the effect is irrelevant since that should already be accounted for by whatever test you do.
Anyway, I don't even think false positives are the right way to think about this. The framing should be continuous, not binary. The goal is to maximize success, not maximize the odds of picking the better of A or B.
A Bayesian analysis tells you Bayesian things: specifically, this is the most reasonable conclusion one can draw from this data, right now. A frequentist analysis also tells you things, but they are different things. Specifically, frequentists are concerned with...frequencies: if we were to run this procedure many times, here's what we can say about the possible outcomes.
There's this persistent meme that all Bayesian methods protect you from multiple comparisons/multiple looks problems. That's not really true. Bayesian methods don't offer any Type I/Type II error control--and why would they, when the notion of these errors doesn't make much sense in a Bayesian framework?
You can certainly use Bayesian methods to estimate a parameter's value. However this you cannot repeatedly test your estimates--Bayesian or otherwise--in a NHST-like framework and expect things to work out correctly.
"It's totally reasonable to roll your own cryptography, as long as you're an expert in the field and do it correctly".
The rules of "Don't Stop And Peek" is general advice that is given out because the vast majority of people that conduct these kinds of trials are not statisticians, and will not do the math. Those that attempt the math will likely do the math wrong.
So, you're totally correct that if you know exactly what you're doing as a statistician and apply the math correctly, it can be possible to peek at results while still drawing a conclusion. This ignores the fact that, for most humans who are receiving the advice, this is terrible advice.
So, general rule: "Don't Stop and Peek". Advanced rule: "Unless you are a trained statistician, Don't Stop and Peek".
He ran some simulated A/B tests using my Bayesian A/B testing technique (which is now powering VWO). He showed that while peeking does ensure that loss < specified threshold, if you don't peek the loss will be even lower.
So although peeking is still valid, that validity does come at a cost.
Of course there's the generic Bonferroni correction where you just divide the significance threshold by the number of tests, but you'll end up with a lot more false negatives (where you just miss the result entirely), and if you want to keep running the test for arbitrary lengths of time it becomes even more difficult (you'll need to keep lowering the threshold). Then again this does give strong guarantees and might work well enough if you don't check the results often.
Basically to get the best results you'll need to balance the false negative rate against the false positive rate and the expected length of the test, which is a rather complicated trade off. But I expect someone will have done at least some of the calculations for simple A/B testing.
"You can perform power calculations by using an online calculator or a statistical package like R. If time is short, a simple rule of thumb is to use 6000 conversion events in each group if you want to detect a 5% uplift. Use 1600 if you only want to detect uplifts of over 10%. These numbers will give you around 80% power to detect a true effect."
These numbers assume approximately 10% of changes give true uplift (aka actual improvement). If the incidence of improvement is rarer you would need more than 6000 events to identify a 5% increase.
(OT: that term sounds like cult language -- you have three chances to achieve true uplift)
FYI, the difference with Google Analytics and most other A/B platforms is that GA uses multi-armed bandit. It adjusts the proportions according to the results. This is nice because it effectively ends the test for you, and you don't have to stress about suboptimal success rates just for the purpose of experimentation. https://support.google.com/analytics/answer/2844870
Yes.
> The larger the relative difference, the fewer events needed.
Wrong. That's the common mistake that is being addressed.
The relative difference could be anything at any point in time. You always need a lots of events to be somewhat confident that there is really a difference.
Short version: If you want X% confidence on a test, there are maths formula to say that you need to monitor N events, at least. The formula doesn't depends on how large is the difference (which you don't know anyway).