Did I miss any reasons?
Did I miss any reasons?
Very good article by Warren Buffet that touches on the same issue: http://www.tilsonfunds.com/superinvestors.html
[0] http://uncertainty.stat.cmu.edu/ Chapter 12.
In practice, if you fix p=0.05 and increase your n, the probability that you will find a statistically significant result often increases because your power increases, and in many situations, the probability that the null hypothesis is true is close to zero. (Andrew Gelman uses the example of asking whether there are significant differences between voting patterns of men and women.)
On the other hand, effect size estimates become more accurate as the sample size increases. This mitigates the above issue, provided you actually report your effect size. It also means that small sample studies that report statistically significant results are more likely to overestimate their effect size, which is especially problematic if you are applying null hypothesis significance testing when you know the null hypothesis is false.