Hacker News Reader

Comment by cl42 | Hacker News Reader

Thanks for sharing this. In general, it's a real shame how few people know how to interpret and use P values correctly. We work with a lot of businesses who ask us to compare populations (e.g., through A/B tests) and people either (a) don't care about the significance between population differences, or (b) are irrationally attached to P values.

Case in point #1: debating whether a P value of 0.051 versus 0.049 is a major difference in the significance test.

Case in point #2: a P value of <0.001 but with extremely low differences in means between populations. With enough data, everything is significant!

End rant. :)

View on HN

> are irrationally attached to P values

This is actually a terrible problem. Most people, I think, do this because of nervous cluelessness: they are sure there is some theory that says what it means, but they aren't sure they understand it, so they insist on following some strict rules, whether they make sense or not.

I think the idea is that if you follow a strict rule given in a textbook (even if wrong or inapplicable), that makes people feel safer than if they try to figure the rule out for themselves. At least if they follow a rule, they aren't personally responsible for the outcome.

It can sometimes be quite difficult to explain why a certain rule should not be applied, because the same reasons for why they didn't understand the rule in the first place also make it difficult to understand explanations given by someone else.

> so they insist on following some strict rules, whether they make sense or not.

I took my bachelor in business and we were always told to make rational business decision. That is, informed decisions, based on data.

As for actual statistics, we were told 2 rules of thumbs: You need at least 30 people in a group before you can make meaningful statistics and it is significant if P < 0.005

Unfortunately that advice is hugely misleading. The reason is that the p-value essentially tells you how repeatable your analysis will be, but doesn't say anything about how beneficial making the decision might be. Second, there is a relationship between how large the difference between outcomes are and how many people you need. So if taking your conversion rate from 10 to 15% would be really helpful to your business, you might need say thousands of people to notice that difference statistically.

> The reason is that the p-value essentially tells you how repeatable your analysis will be

Your statement contradicts the introductory paragraph of the article:

> "...the outcomes of these tests are mistakenly believed to yield the following information:... the probability that an initial finding will replicate; ..."

The p-value is the likelihood that the observed effect is due only to chance, not a measure of the repeatability of a result.

Ah yes, you're right! I made a statement about likelihood!

This was meant as an example of how crazily statistics is taught and why people end up just following strict rules.

Amen for #2! It's like the entire field of biology is an endless quest for low p-values, regardless if the hypothesis is even interesting. It's like people are not interested to think, they just want to publish something significantly differnt.

haha, Sociology as well -- especially now that the web provides huge amounts of behavioral data.

I much prefer how machine learning folks tend to approach predictive accuracy, though I guess that's not quite the same as understanding relationships between specific variables while controlling for others.

There is a notion of 'feature importance,' which especially comes up in decision trees and random forests, giving a notion of how much a particular feature contributes to the overall prediction. It seems like combining predictive power with feature importance would be an interesting alternate route to demonstrating important correlations. (For example, maybe a model predicts lung cancer with 90% precision, and 'is_smoker' has a 80% feature importance.) Of course, these importances depend a lot on the other features used by the model! If you include a lot of junk features and/or exclude other important features, the importance of your pet feature will shoot up.

Hmm, interesting -- I never considered the idea of including junk features to bias model's preferences of whatever theoretically ambiguous idea you're trying to promote. That's actually brilliant.

Shameless plug; I'm a co-author of a method that leverages adding artificial junk features and removing original ones that are likely nonsense to approximate the set of all features that are relevant to the problem (rather than standard make best model, which may be pretty deceiving). https://m2.icm.edu.pl/boruta

Shoes reduce cholesterol.

And aren't ice cream sales dependent on murder rates?

Did you just discredit Freakonomics?

> they just want to publish something

What's going on in the academia is a classical example of perverse incentives. If your career depends on churning out soul-crushingly bad or insignificant papers that fulfill some superficial criteria of "scienceness" then that's what you'll do. It's dangerously close to cargo cult science.

Addressing that first point is "The Difference Between 'Significant' and 'Not Significant' is not Itself Statistically Significant" by Andrew Gelman and Hal Stern.

PDF link: http://www.stat.columbia.edu/~gelman/research/published/sign...

Nitpick: while that paper does mention the basic idea that there's no significant difference between 0.051 and 0.049, it's more about another common mistake: testing the difference between two groups by performing two separate statistical tests and then comparing the outcomes (a <> b), rather than the correct approach which is to do a single test for the difference between groups (a - b <> 0).

One of my favourite go-to phrases in health care is: "Statistically significant, but a clinically insignificant difference."

I love it! That's actually a really good way of looking at it, right? Unfortunately for researchers, I'd argue that "Statistically significant, but a clinically insignificant difference." is equivalent to "Theoretically useless, and technically irrelevant." :)

In quant finance, my colleagues and I say, "OK, it's statistically significant, but is it economically significant?" I love that we have parallel sayings in totally different fields.

Right, you're describing statistical significance as it relates to practical significance.

Reply on news.ycombinator.com