Hacker News Reader

The ASA recently published a new statement which is more optimistic about the use of p-values [1]. I myself also think that correctly used p-values are in many situations a good tool for making sense out of data. Of course, a decision should never be conducted on a p-value alone, but the same could also be said about confidence/credible intervals, Bayes factors, relative belief ratios, and any other inferential tool available (and I‘m saying this as someone who is doing research in Bayesian hypothesis testing methodology). Data analysts always need to use common sense and put the data at hand into broader context.

[1] https://projecteuclid.org/journals/annals-of-applied-statist...

Is the nuance here that the ASA is OK with p-values but not OK with the rhetorical phrasings around statistical significance? My take is that it is easy to casually misinterpret or misrepresent statistical results because of how fuzzy these language around it all is. Phrases like "statistically significant" imply a certain kind of causality to the reader, when the actual rigorous claims are very specific and nuanced. Moving away from such soft phrasings might mean people have to stick to precise and narrow claims, whereas the normalization of soft phrasings makes room for bad claims or bad interpretations.

I read through the first link you posted and couldn't find any ideas about what we could use instead of p-Values.

Statistical tests are a very useful and objective method of determining whether the outcomes of one thing/activity are more desirable than another when applied correctly.

Some solutions could be to set a higher bar for statistical analysis education. Or perhaps a more thorough statistically focussed vetting and peer review process for published material?

.. and after reading the second link I note that it includes commentary around the insurmountably difficult challenge of replacing the current paradigm. While not offering a solution, it goes on to provide some great advice for how to improve the quality of statistical research.

Following all of the advice to the letter would make it almost impossible to conduct valid research.

Maybe this is where economists have the upper hand? Just develop some highly abstract mathematical models representing the topic you're interested in, and abstain from rigorously testing them.

Bayesian hypothesis testing, including Bayes factors, might be more useful.

The second link addresses what should be done instead. It's not entirely satisfying as there isn't yet consensus for a replacement, if such a thing is even possible.

It's worth pulling the principles from the ASA's statement [2] as well:

  1. P-values can indicate how incompatible the data are with a specified statistical model.
  
  2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
  
  3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
  
  4. Proper inference requires full reporting and transparency
  
  5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
  
  6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

The basic criticism one of brittleness - that unless very carefully planned, executed, and interpreted, p-values from hypothesis does not support the claims some would like to be on their results, and that meeting the first condition is so difficult that the technique should not be recommended. One _should_ look for 'significant' results, but using measures that align better with colloquial understandings of significance i.e. with how users are misinterpreting p-values now.

> P-values do not measure the probability that the studied hypothesis is true

So, what’s the best way to measure the probability that the studied hypothesis is true?

The argument against p-values is part of the argument against any bright-line single-number rule for identifying truth. The job of the researcher is to demonstrate (at least) that

1. there is (isn't) an observable difference between groups of interest

2. the difference is (not) attributable to hypothesized causal mechanism i.e. the (absence of a) difference isn't due to random variation in the observed sample i.e. the difference would be observed by a independent replication of the same analysis/study

3a. the difference is not explainable by other factors that vary between the groups, observed or unobserved

3b. the difference is not artificially inflated (suppressed) by the statistical choices

4. the difference is large enough to be practically relevant.

and so on

If the degree of certainty of statements about the difference can be characterized by a single number at the end of the process, great! But the goal should be a convincing, wholistic story, not the single number.

I share your concern, and I worry that we'll find this battle continues 20 years from now.

There are many possible things that can go wrong with a P-value and I'm not a statistician, but things I look for in data are the structure/distribution of the "noise" and any correlations seen within the "noise" and the "signal". That helps you build a signal and noise model. Assuming that all your noise is inherently uncorrelated gaussian, is a pretty strong model assumption.

Use Bayesian statistics [1] :-)

[1] https://en.wikipedia.org/wiki/Bayes_factor