Don't calculate p-values without understanding them
scottbot.net
scottbot.net
That said, if you need to use p-values in A/B testing, you might want to read http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigo... to get a procedure that gives always valid p-values, and then http://elem.com/~btilly/ab-testing-multiple-looks/part2-limi... for a more practical alternative and the caveats. (I still intend to return to the series, but not for a bit more.)
If you assessed differences in average human height between two cities based on a million datapoints, you're pretty darn likely to find a difference that's statistically significant but not really important or meaningful (like a .1mm difference in average height).
The above is why the American Psychological Association says "reporting and interpreting effect sizes in the context of [p-values] is essential to good research".[1] It's also why Statwing always report effect sizes along with p-values for hypothesis tests.[2]
(For clarity: effect size is best presented in readily interpretable, concrete terms like height or whatever unit you're using. If that's not possible because you're comparing ratings on a 1 to 7 scale or something, or if you want to compare effect sizes across different types of analyses, there are specific metrics of effect size).
Another common cause of finding significant p-values when you shouldn't is looking at many metrics (or the same metric at many points of time). Eventually you get (un)lucky.
However if you do the stats correctly, and the statistical model (which usually assumes independence) is accurate, then a p-value of under 5% will only happen 5% of the time. No matter how big the sample size is.
In my totally made up example, there's really a .1mm difference. And maybe to someone that's meaningful, so it's worth reporting. But it's not a big enough effect to say "When I see someone on the street I can look at their height and know which of these two cities they're from."
And my peeve is that people will see a statistically significant difference and think that that alone makes it an important finding, versus a very real finding that might not actually matter that much.
Or, related, it's bad to see something with a p-value of .15 and a large effect size and decide that since it's not significant at .05 it's not interesting. Since that combination typically means that you have a too-small sample size, the best interpretation is probably more like "This might be interesting. Looks like we don't have enough data to tell if there's a relationship here, but it probably deserves another look."
(Edited to add that last thought, and clarify the previous one)
I'm halfway through a major revision of my guide to statistical error that discusses this topic:
http://www.refsmmat.com/statistics/regression.html#truth-inf...
>"If it’s under 5%, p < 0.05, we can be reasonably certain that our result probably implies a stacked coin."
By itself, a p-value is NOT enough to imply that the null hypothesis is false. In fact, if I flipped a regularly looking coin and saw 7 heads, I'd still be very confident that the coin is fair, because weighted coins are so rare. Later, the article correctly warns:
>P-value misconception #5: "1 − (p-value) is not the probability of the alternative hypothesis being true (see (1))."
P.S. I think the weasel words in the first quoted sentence, "reasonably certain" and "probably implies," show that the author is at least subconsciously aware of this logical error. :)
is by Uri Simonsohn, a professor of psychology with a better than average understanding of statistics, and colleagues who are concerned about making scientific papers more reliable. You can use the p-curve software on that site for your own investigations into p values found in published research.
Many of the issues brought up by the blog post kindly submitted here and the comments that were submitted here before this comment become much more clear after reading Simonsohn's various articles
http://opim.wharton.upenn.edu/~uws/
about p values and what they mean, and other aspects of interpreting published scientific research.
I'm skeptical they would be used nearly as much if they were properly understood.
> you can just pretend you went into your experiment with different halting conditions and, voila!, your results become significant.
You can misrepresent your results regardless of the underlying statistic. But it's no easier to lie about p-values than to lie about any other statistical procedure.
Anyway, the post seems to be more about hypothesis testing than p-values per se.
"It is for this reason that I’m trying desperately to get quantitative humanists using non-parametric and Bayesian methods from the very beginning, before our methodology becomes canonized and set."
:)
The "the design of the experiment shouldn't matter so much" assertion is usually followed by an appeal to the likelihood principle and a claim that frequentist estimation is misguided. If that's not what you had in mind, apologies. I've never seen it coupled with a claim that the frequentist would then misrepresent their experiment...
If the quoted statement is your goal, I think a more convincing argument is, "we often want a more nuanced way to express uncertainty than classical tests/confidence intervals give us, and... LOOK! We get that for free using Bayesian principles."
As an aside, running out of funding seems like it should give you the same results as a predetermined sample size, as long as the funding isn't conditional on getting interesting (i.e. statistically significant) results, but I'd need to actually do the math to be certain.
My goal is to make people aware of the various stats out there, their benefits and pitfalls, and let people choose whatever is the most appropriate for their needs. Those choices need to be informed, and given that most introductory stats starts with p-values and seems to teach them wrong, that's where this post is aimed.
Regarding your aside, the universe of possible observations in a given experiment assuming 100 trials may be very different than the universe assuming trials until we run out of money, which happened to fall on 100.