Hacker News Reader

Comment by p-e-w | Hacker News Reader

The statistical interpretation of observations is so subtle and complex that it's a good idea to assume that any publication from the empirical sciences is complete garbage, until you know for sure that a qualified statistician has supervised the process. A semester of "introduction to statistical methods" (which is all the background that most scientists have) is NOT enough.

Imagine a mathematician writing a paper on a medical topic, making all kinds of claims on how things work in the human body – and then that mathematician justifies their expertise by saying "I did a two-week first aid course once, and also, I was really good at biology in school". This is pretty much how lots of science operates when it comes to interpreting results mathematically.

View on HN

I read a survey once, that found that a huge number of PhDs/researchers in the studied sample gave an incorrect definition for what a "95% confidence interval" (/p-value, etc) actually means, and that several popular introductory textbooks defined it incorrectly as well. Wish I bookmarked it.

At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analyses can be redone and flaws can be spotted.

I think you may be talking about "Mindless statistics" by Gigerenzer. He has some surveys about p-values and how radically wrong they are usually interpreted.

>At bare minimum, journals need to require that researchers publish all their data alongside every paper, so statistical analyses can be redone and flaws can be spotted.

Absolutely.

P-values work great when they’re super low, experiments run at a human-scale frequency, and hypotheses are extremely precise in their predictions, e.g. some physics.

If you run an experiment a day and get p < 10^-9, your priors, your multiple hypothesis correction, even your interpretation of p-values approximately don’t matter. Running social sciences experiments with p < 0.05 threshold is where things get weird.

Did you read the article?

>even your interpretation of p-values approximately don’t matter

"Small number means good" is not a sufficient working understanding of p-values for doing science.

But what if it’s really small?

It's completely irrelevant if you don't understand how to interpret it. It is not a number which tells you how correct your hypothesis is.

That is literally mindless statistics. Which coincidentally is the name of the article I talked about. Did you read it?

(In a social science, if your p-value is 1E-5 or something, the most likely interpretation is that you are doing something very wrong)

I did. My statement was hyperbolic. More directly: P-values are more resilient to misuse at their extremes.

probably this article: Hoekstra, R., Morey, R.D., Rouder, J.N. et al. Robust misinterpretation of confidence intervals. Psychon Bull Rev 21, 1157–1164 (2014). https://doi.org/10.3758/s13423-013-0572-3

another good article on misinterpretation of p-values and confidence intervals is: Greenland, S., Senn, S.J., Rothman, K.J. et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 31, 337–350 (2016). https://doi.org/10.1007/s10654-016-0149-3

While I agree on the data point, it would kill so much research. It is bad that a lot of research validation basically comes down to "trust me guys", but with data being both very valuable and often times highly sensitive, it can be really difficult to just publish the data along with the research.

A decent compromise would be to at least require meta-data to sufficiently exclude some flaws. A different approach could be to have researchers document and publish the process of th research, similar to a git-repo with the main branch being completely off limits to history-rewriting.

On the other hand it seems there's also a lack of testing subjects. It's already frequently pointed out how medical results might not represent everyone. I would also assume that e.g. pharmaceutical certification processes do apply more sophisticated statistics.

The ugly truth is that lots of "science" being done today isn't actually science – it's a performance art that superficially imitates certain behaviors that are associated with real science.

And how could it be otherwise? There are nearly 10 million scientists in the world right now. And all of them are pushing out papers as fast as humanly possible. There isn't anywhere near enough statistical brainpower available to quality-control all of that. Not to mention that most people with sufficient expertise in statistics have better things to do than micromanaging science grad students who have a hard time comprehending Bayes' theorem.

Bayes theorem for laymen? I read an AI book and it talked about how you can predict the cause from the effects given certain probabilities.

Ah, I see you have not met a biophysicist.

Reply on news.ycombinator.com