“Psychological Science” backs away from null hypothesis significance testing
andrewgelman.com
andrewgelman.com
From the article:
>"One reason so many of these Psychological Science studies are so dead on arrival is that they hinge on noisy measurements in uncontrolled, between-subject designs. That puts you right here, and no amount of preregistration or fancy statistics is going to solve your problems."
It must be great to work in a field where you decide what the result is. You can't do that in mathematics unfortunately. It must be why all these mathematicians are so bitter with social sciences.
You do get noisy measurements. That's why you need to pay a lot of attention to your confidence intervals and standard errors - they give you a sense of your signal to noise ratio. Not all psychologists and social scientists are good at that, and that's a problem, but IMO it's really not _that_ difficult to distinguish babies and bathwater here.
And I think Feynman's comment about between-subject designs is outright dangerous. You do not necessarily reduce your noise by switching to within-subject designs. You just move your noise to a place where it's much more difficult to measure, and possibly create a lot of endogeneity that you can't deal with except by throwing out a bunch of assumptions and hoping you're right. Sure the numbers that spit out of your equations start looking more tidy, but that doesn't mean your results are more trustworthy.
But it throws back to Feynman's speeches on Cargo cult science, what is science and what isn't. The idea is basically the same, we can describe what good science is by the rigor put in the experimental process, but this is not something that can be captured simply by the p-value.
So...a jury?
>"All too often I feel like I’m seeing the attitude that statistical significance is a win or a proof of correctness, and I think this pushes researchers in the direction of going the cheap route, rolling the dice, and hoping for a low p-value that can be published. But when measurements are biased, noisy, and poorly controlled, even if you happen to get that p less than .05, it won’t really be telling you anything."