Imagine you have a purchase form that converts at 0.1%. You test a new change, and with the first 22 users (11 in each group), if the control group randomly gets a conversion, it will have AT LEAST a ~9% CVR now. Okay, maybe you ignore that data, since you can just compare to historical data. Whatever.
Even if the new form is dramatically better and has a 30% CVR -- There's a 3% chance it would show a ~9% CVR also, and about the same that it would show a 0% CVR. Similarly, there's about a 6% chance it would show a 91%+ CVR -- skewing your results just as badly in the opposite direction. There's only like a 30% chance you get close to the "true" CVR.
Unless these hydroxycloroquine is close to 100% effective (and that doesn't seem to be what any of these studies say) I just don't understand how there's any confidence with small sample sizes.
Does anyone have a link to how this works? I've asked a few friends in medicine, and they haven't given a good answer. Haven't found anything on Wikipedia. Probably don't know what to search for.
If an A/B test took a form from almost 0% conversion to almost 100% conversion, you might only need 11 users to show that. But even in that case, the devil is in the details. Were your users really a completely random sample from your population, etc?
The other side of the coin is, can we say these are non-interesting treatments with only 11 subjects? Or are the studies too underpowered to say much of all?
Depending on what we mean by "back to life", a trick is in fact much more likely than something thought to be biologically impossible.
If we eliminated the possibility of a trick -- that is, proved beyond a doubt that 11 people were dead, and then alive -- I would absolutely not be ready to attribute it to some new treatment. Natural law is being violated here! I would instead start checking for other possibilities, like:
- Are other people spontaneously reanimating around the world, or just in this study?
- Did these 11 people have anything else in common? Extraterrestrial origin, perhaps? Unusual religious beliefs?
- Hang on, let's double (triple, quadruple, etc.) check that they were really dead. And then go back and check again.
- And by all means, start a proper clinical trial of the new treatment while we're trying to understand how resurrection is even possible.