Relevant XKCD: https://xkcd.com/882/
Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. Only publish positive results, and someone will eventually publish a false positive result without fraud.
Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. Only publish positive results, and someone will eventually publish a false positive result without fraud.
You need different samples. There isn't a statistical trick that gets around this.
For example: let's say there's a cancer with 20% survival rate. You test a treatment with 25 experimental and 25 control patients, 40% in the experimental group survive[1].
You can analyze this with a bunch of statistical methods. You can ask different questions about the patients, focusing on well-being rather than simple carcinogenic remission. But ultimately, the thing that happened in this study is that 50% got better and no fiddling with numbers or changing the questions you ask is going to change that underlying phenomenon. You can check for blood markers of cancer: you get 50% have no blood markers. You ask them questions about how they feel: you get 50% feel better. You body scan the area where tumors were: you get 50% no longer have tumors.
You have only tested one phenomenon in one sample, and that essentially amounts to 5 people getting better.
[1] I know this is not how cancer treatment studies work exactly, this is a simplified hypothetical.
Eventually if we (the public) are lucky someone else in the field will disagree and run the trial again, which is how you get the alt text.
Particle physics still uses five sigma as the significance threshold.
I think most people correctly intuit that this is actually a type of very pernicious fraud.
This seems like such an obvious problem in the way science is currently done. Are people so focused on their own individual fields that they aren't thinking about and fixing such glaring meta problems?
Consider: "Hey, look, I went on top of tower of Pisa and threw down two identically shaped balls, one iron and one wooden. They dropped at the same time!"
The above is the expected result and would only be interesting if the result is different from expectation. Now, if 1000 scientists did this and each published the confirmation of what we knew would happen then who would read that? But, if one scientist said: "I tried and they drop at different times!" that would be different. The 999 scientists would then try to replicate again and then the papers of the 999 would be interesting again.
That would be true if p were the probability of null hypothesis being true given the data observed, but that’s not what it is.
https://eurradiolexp.springeropen.com/articles/10.1186/s4174....
This is the sort of thing I used to do when I was younger, and looking back, the reason I did it was because I was basing my sense of self-worth in being smarter than other people. Ironically, this made me dumber, because I was less open to the possibility of being wrong and therefore was slower to learn. And, I found doing this made people dislike me.
I think you're interpreting that sentence as talking about one false positive out of 20 positives? Which would be a very incorrect statement.
But it says one false positive out of 20 experiments. That's a valid statement. (Though it does depend on the proportion of negatives you're getting.)