Statisticians want to abandon science’s standard measure of ‘significance’
sciencenews.org
sciencenews.org
Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way.
You can actually compute p(H|x) if you use Bayesian stats.
Typically people doing research have prior information about what they are researching. E.g. previous studies have found effect sizes to be in some interval.
You can fallback on an uninformative prior or other tricks if you really want to model the idea that you know literally nothing about what you're researching. But that should be very rare.
Our 0.05 p-value limit is effectively just answering that question with a fixed 5%, no matter how ridiculous the proposition is.
You can't "compute" it for any useful meaning of the word "compute". You can estimate it intuitively, or you can try to look at how many pre-registered unpublished studies or null results have been published. Otherwise there's no way to even consider getting a grasp of what P(H) would be.
That's called picking a prior. Usually, pick the prior that maximizes the entropy on the given interval (zero prior knowledge).
For those reading and confused. Computing: p(H|x) = p(x|H)p(H)/p(x)
requires p(H) which the parent is suggesting is impossible to grasp.
If we want to know the probability that a coin is biased, we can assign probabilities to each hypothesis. For some people, p(bias=1/2)=1 (Dirac distribution), others might argue that p(bias=x)=1 for x in [0,1] (uniform distribution). Others might argue its some Beta function centered around 1/2. I believe what the parent is suggesting is that choosing which original belief we have in the system is a matter of philosophy, not computation.
Essentially, we want to get p(H0|x), but we need Bayes Law to get this from p(x|H0). But we need some notion of what priors to use. This is of course impossible to actually get, but if we published null studies then it would allow us to estimate it with better confidence. Alternatively we can ballpark it by saying how unexpected the result is, and how long the effect being explained has been studied.
The "common-sense" version of this is intuitive, that "extraordinary claims require extraordinary evidence". The more unexpected a result is, the lower the p-value has to be to be convincing.
The epidemiological version of this shows up in things like the Bradford Hill criteria [2], which include significance of association but also attempt to bring in plausibility.
[1] https://statmodeling.stat.columbia.edu/2015/03/02/what-hypot... [2] https://statmodeling.stat.columbia.edu/2015/07/03/why-should...
It seems like something that should be taught alongside math from elementary school, not something you can maybe get an elective in in high school or college.
I actually found statistics pretty fun and eventually we learned to use R for it. Though I ended up missing 1 lecture and it almost screwed me over for the rest of the class. I found a Kahn academy video though that covered it and was able to catch up. There really is a ton of information to learn. It would have been nice to have at least touched on it a little bit in highschool beyond learning about means and the tiniest bit about standard deviations.
I know a lot of people have trouble thinking clearly and correctly about probability and statistical inference.
But do we know if, practically speaking, that can be addressed by a modified educational curriculum?
Or are these concepts that would take an extraordinary amount of effort for many persons to understand well?
[1] source: I never saw it as a student, and I've consulted with math teachers (well, administrators, about math) in multiple states, and I've never had it come up there either.
The problem is that we are imbuing the words "statistical significance" with a whole bunch of math. This would be fine, except for the inconvenient fact that people also want to use the word "significance" as it is defined in English.
It is not only possible, but likely that people will be producing results that are insignificant but statistically significant. I mean seriously, if I tell my boss that the results are statistically significant, how do I expect him to understand that the results might reasonably be insignificant? How do I expect anyone non-technical to take that sentence seriously? You lose all credibility pretty quickly when people start saying that the result is significant except it doesn't matter.
This is even more stupid than calling complex numbers "imaginary" and "complex". Those names are just arbitrary. Statistical significance is overloading words that are likely to come up with meanings other than what they are expected to mean.
Anyway, this is the problem that things like this article are going to. People are treating significance values as though they are significant. They aren't significant, they are statistically significant. It means something different to significance.
For example; if I run an experiment 20 times and get a p value consistent with a 5% chance in one experiment, it is statistically significant but the result is obviously not significant in the English-language meaning of the word.
In fact, failures to reproduce "landmark" experiments have low significance (statistics word) because they find a result that is highly probable if the null hypothesis is true. But failures to reproduce have high significance (colloquial English word) because they indicate that widely-held beliefs are wrong. So in some of the most important cases, the statistics and colloquial English meanings of the words are exactly the opposite.
I don't know if there's a good solution to this, though--it's hard to change organically-emergent trends in language even if we did come up with a better term.
If you look at research on carcinogens, it is frequent to see something like "it is statistically significant that eating chemical X increases your risk of cancer" but when you read on, the magnitude of the effect is "increases risk by .0023%, for a cancer that already has a .6% incidence in the population".
My point is that people who are not statisticians (or people who use statistics), tend to be concerned with the magnitude of an effect, and frequently misconstrue the statistical significance for that magnitude.
But these concepts aren't clear to the people using the data. I've seen people use statistical tests done in batch without correcting for multiple testing (one of my regrettable tasks), and flat out ignore any differences that weren't statistically significant. They thought it meant "no difference."
With population-level data (e.g., Census counts or hospital records), I sometimes wonder if statistical significance has any value. It's a tool to make sure we don't accidentally claim a difference where there isn't one. But I can guarantee you that nothing worth measuring is exactly the same for men, women, whites, blacks, young, old, whatever. It'd be amazing if the difference was 0. So statistical significance ends up being an obfuscated way to state the sample size.
My decision to select, implement, go, or no go cannot itself be a probability distribution.
The problem is that people want an answer to the question, "Here is a pile of data, what should I believe?" But mathematically, the proper role of data is to modify existing beliefs, and not to dictate beliefs.
Statisticians can spend forever explaining this. But instead have gone with the cop-out of asking a question that is confusingly similar to the one that people want to ask. It is popular exactly because it is so easily misunderstood. You can give any number of lectures on what it actually means - I guarantee that it will be misunderstood.
Worse yet, p-values are sensitive to particulars of experimental design that logically should never matter to inferences. My aunt and uncle provide a classic example. They wanted a son and a daughter. They had 6 sons then a daughter. Are they biased towards one gender? The p-value for this question is .03125 (we'd see this much evidence against with all boys, all girls, last girl, last boy, so 4/2^7). But if I instead said that they planned to have 7 children, and had 6 sons then a daughter, the p-value changes to 0.125 because the one child out could have been anywhere in succession. But in Bayes' formula, their plans if something else had happened cannot ever affect how we adjust our inferences.
(This example is not made up. When Lorna found that they had a daughter, she told Bill to get a vasectomy from the delivery table. By another crazy coincidence, their 7 children were also born on all the days of the week.)
Its not just about p-values or the likelyhood of making a particular type of error by agreeing with a thesis.
There is a fundamental misunderstanding that most people, and even many scientists make with regards to the philosophical interpretation of 'truth' using the scientific method.
By definition, scientific 'truth' is always fallible, and the vast majority of people have a very hard time dealing with this concept. A better explanation or additional data, or better data is always possible and needs to be possible for science to not be dogma. This is fundamentally different than how people interpret truth and understand what the word means. All scientific truths are replaceable.
That's not how statistics works. Frequentist p-values and Bayesian inferences are both entirely dependent on the mathematical model in question. "Their plans if something else had happened" are a key part of the model. What you're describing at 2 entirely different experiments:
1) A couple has children until they have 1 of each gender. they have 7 children. How likely is a result this extreme (or greater)?
2) A couple has 7 children. They have a gender ratio of 6:1. How likely is a result this extreme (or greater)?
The probability of event A given that we observed B is the probability of A and B happening divided by the probability that B happened. Mighta, coulda, shoulda but didn't doesn't enter into it and can't affect the result. The fact that frequentist statistics does care and shouldn't is one of the major criticisms that Bayesians offer.
If you think you understand statistics and don't understand this fact, then you do not understand statistics as well as you think you do. But you can be pardoned. Most statistics classes are too busy cramming statistical tests into student's heads to bother them with bothersome facts about where the cracks in the foundations are.
So true. That said, here's a great interactive explaining the p-value: https://www.jwilber.me/permutationtest/
In my work as a data scientist, 99% of the time this is not even relevant. We have huge sample sizes and most of the p-values we see are <.0001, meaning there is SO much data that even a very very small effect size can be found "significant."
The question then: does a mean of 20.02 being significantly different from a mean of 20.03 over 10 million patients, is that actually important to us?
[1] http://computationalimagination.com/article_practical_signif...
The big problem is that since we've already long since entered the world of big data you can now find spurious correlations, similarly strong, but constrained to variables that are plausibly connected. For instance what if this correlation was between spending on science/space/technology and an increase in the number of people pursuing STEM fields? People would immediately just accept it without question, even though it should in theory be held to the same scientific rigor and critique as one that challenges our biases. Science should not be a glorified exercise in confirmation bias. Spurious correlations don't mean you have a biased sample or that such correlations don't exist - it simply means that correlations have no direct relationship.
538 has a great little interactive p-hacking game you can play, with real data. [2] Ultimately any single datum that tries to legitimize a piece of research is going to be gamed. Science should be judged as it was in its 'glory days' - by the logic, predictability, and falsifiability of the ideas presented. Anything that steps outside these bounds should be treated with the most extreme of prejudice. It may be legitimate but the great burden of proof is on the presenter, and that doesn't come down to p-hacking up a 0.00001 correlation and calling it causal.
If we required scientists to be good statisticians there’d be far fewer scientists.
P hacking isn't always done on purpose. It's misunderstanding. That's probably why it still passes peer review. There's also tons of incentives that encourage this behavior. These are the problems. An arbitrary mark to meet makes people weak at statistics not understand their data as well. I'd rather more accurate data than more scientists. More workers doing poor work isn't useful.
Also, you can Bayes hack. Bayes helps, but it doesn't address the underlying issues.
In my view, understanding p-values won't help. Feeding data into a formula that produces a p-value doesn't account for things that can go wrong with experiments: Uncontrolled experimental conditions, biased sampling, hidden correlations, and simply non-ergodic systems. It is possible that some systems can't be controlled to the point where it's reasonable to start looking for real effects.
I suspect that if the results were any good, the precise interpretation of the p-value wouldn't matter. Physics has lived without an agreed-upon interpretation of quantum theory for a century.
People that work with computers often are slightly better about reasoning about probabilities because they often deal with high frequency events. A 1% error rate across 10 million events a day leads to 100,000 errors. If those events are transactions, payments, file uploads etc, a 1% chance starts to look very common. But even computer people are not immune to incorrectly reasoning about probabilities. I suspect there is some fundamental limitation at the cognitive level.
You might see p=0.00001 but if the size of the effect irrelevant, then the statistical significance of the relationship is still not something I care about. Great, orange juice shrinks tumors by 0.000001%. I'm still going with chemo, thanks.
To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p<.05, but it won't be meaningful.
We were taught that "effect size" measures were crucial to understanding and interpreting results. If you have p<.05 but a tiny effect size, you're likely not seeing a meaningful difference.
Just to clarify. Any study with a sufficiently large sample will find a real effect even if the effect isn't clinically significant.
I don't think that is right. Think about tossing unbiased coins. The probability that you will get an improbable (p < 5%) result is exactly the same no matter how many or few coins you toss.
(Premise) If Tracy is an American then it is very unlikely that she is a US congresswoman; (Premise) Tracy is a US congresswoman; (Conclusion) It is very likely that Tracy is not an American
(Premise) If the H0 is true, then is it very unlikely that I will observe result X (Premise) I observe result X (Conclusion) Therefore, it is very likely that the H0 is not true
A simple example of this can be shown with the following pair of tests:
Testing for a fair coin:
- Null hypothesis is you have a fair coin
- You observe 100 heads in a row
- Given a fair coin, it's extremely unlikely to observe 100 heads
- Therefore it's not a fair coin
Testing whether a person "Bill" is an American:
- Null hypothesis is Bill is an American
- You observe *Bill is a US congressman*
- Given Bill is an American, it's extremely unlikely to be a congressman
- Therefore he's not an American
The problem isn't that P=.05 is an arbitrary measure of significance. The problem is that only publishing significant results is a bias against the null hypothesis.
Let's say you're doing a study of flipping coins. The null hypothesis is that the coin is evenly weighted. If the null hypothesis is true, when you flip a coin once, it will come up heads with P=.5. If you flip the coin twice, the null hypothesis is that both flips come up heads with P=.25. The probability of all coins coming up heads is P=0.125 for 3 flips, P=0.0625 for 4 flips, and P=.03125 for 5 flips. So if we flip a coin 5 times, and get heads all 5 times, we can conclude that the coin is weighted in some way with P=0.03125.
Let's say all the major journals of coin flipping only publish results with the high significance of P<0.05. Alice flips a quarter 5 times and gets 3 heads and 2 tails, and nobody will publish her study because it has P=.3125 (see [1] for an intuitive explanation of how this was calculated). Bob flips a quarter 5 times and gets 2 heads and 3 tails--again, no journal will publish him. Catherine, David, Ellen, Frank, and Geri all perform the same experiment, most of them getting groupings 3:2 result ratios, some getting 4:1 result ratios, but nobody getting all heads or all tails, just as one would expect given the null hypothesis. And the journal editors tirelessly send out rejection letters to all their studies.
Now somewhere down the line, Robert flips a coin 5 times and gets 5 heads. This result has P=.03125, which meets the requirement of P<.05! He sends it to the American Journal of Coin Flipping Studies (AJCFS), and they are very excited to publish his results! Nature and Science magazines do front page pieces with headlines "Quarters Found Heavy-headed" and "Washington Shows His Face" respectively. A casino hires Robert as a consultant for the design of their coin-flipping games. His quarter-flipping study is cited in the abstracts of two dime-flipping studies and a half-dollar flipping study. During the trial of a murderer who placed quarters tails-side up on his victims, Robert is called as an expert witness to say that the coins were placed there, not flipped there.
A week later Sally flips a quarter 5 times and gets 2 heads and 3 tails. She sends the results of her study to the AJCFS noting a failure to reproduce Robert's result, but her study is rejected because it has P=.3125.
Now, if you survey the AJCFS and all the other academic journals on coin flipping, you'd conclude that quarters are significantly weighted towards heads. But in fact, the null hypothesis is true: quarters are pretty evenly weighted. Robert's low-P result is exactly what you'd expect to happen eventually if you have enough people perform the 5-quarter-flip experiment--in fact, if a lot of people are studying coin flips, the P of getting a low-P result approaches P=1. But because the AJCFS has a P=.05 requirement, they've created a bias against the null hypothesis, which deceives the public into thinking that flipped quarters are more likely to come up heads than tails.
This is likely the reason why so many fields, most notably psychology[2], are having a replication crisis[3] and a similar effect can be used in P-hacking[4] to bolster results that are essentially fake.
Unlike coin-flipping, fields with replication crises like psychology and medicine have real affects on real people's lives. It's irresponsible and unethical for journals to publish with a bias against the null hypothesis, and adjusting the P-value requirements to another significance requirement, even a less arbitrary one, doesn't fix the issue.
The solution, I think, is for journals to commit to publish studies before the study has been performed, based on the methodology, previous studies on the subject, and qualifications of the researcher. This would mean that many, many studies would be published with null results, and that would be a good thing.
[1] There are 32 possible outcomes for flipping a coin 5 times. If we group them by how many heads and tails, we can calculate a probability for each outcome:
HHHHH 1 result of 5 heads -> P = 1/32 = .03125
HHHHT
HHHTH
HHTHH 5 results of 4 heads, 1 tails -> P = 5/32 = .15625
HTHHH
THHHH
HHHTT
HHTHT
HHTTH
HTHHT
HTHTH
HTTHH 10 results of 3 heads, 2 tails -> P = 10/32 = .3125
THHHT
THHTH
THTHH
TTHHH
HHTTT
HTHTT
HTTHT
HTTTH
THHTT 10 results of 2 heads, 3 tails -> P = 10/32 = .3125
THTHT
THTTH
TTHHT
TTHTH
TTTHH
HTTTT
THTTT
TTHTT 5 results of 1 heads, 4 tails -> P = 5/32 = .15625
TTTHT
TTTTH
TTTTT 1 result of 5 tails -> P = 1/32 = .03125
[3] https://en.wikipedia.org/wiki/Replication_crisis
[4] https://journals.plos.org/plosbiology/article?id=10.1371/jou...
EDIT: Also see roenxi's excellent post on how "significant" means different things in statistics and colloquial English: https://news.ycombinator.com/item?id=20895893
The main article kept saying that p values were not meant to be definitive, but without explaining what was wrong with them. At least, not very clearly.
This comment is a much clearer explanation -- imho -- as to what goes wrong in the industry. I also think it is right to focus the attention away from a particular statistical measure.
I have the opposite problem: my abilities lie more in the statistics/science than the communication--I suspect the only reason that this explanation is being received so well is that I happened across an effective example by pure luck. The probability of me communicating effectively is probably P < .5. ;)
For example, if you dig, you'll find that a lot of the evidence for a correlation between telomere shortening (associated with aging) and processed meat consumption comes from a 2008 study in the American Journal of Clinical Nutrition[1]. They found a P=.006 correlation between processed meat consumption and telomere shortening. But when you look into the study further, they collected data on 47 different food groups.
Let's calculate how unlikely a P=.006 event is if you try 47 times to find it:
A. If you have a probability of something happening once P_1, then the probability of that happening x times in a row, P_x, is P_x = P_1^x. For example, the probability of flipping a coin once and getting heads is P_1=.5, so the probability of that happening twice in a row is P_2=.5^2=.25, which you can verify by enumerating the possibilities (1 in 4 of the following combinations is 2 heads in a row: HH, HT, TH, TT). The probability of flipping 3 heads in a row is P_3=.5^3=.125 (only 1 of these 8 possibilities is all heads: HHH, HHT, HTH, HTT, THH, THT, TTH, TTT).[2]
B. The chances of a P=.006 event NOT happening P' are P'=1-P=.994 (the rule of 1).
C. Combining facts A and B, the chances of a P_1=.006 event NOT happening (P'_1) 47 times in a row P'_47' is P'_47=.994^47=.754 (this is rounded to the 3 significant digits).
D. Applying the rule of 1 again, the probability of finding a P=.006 result if you try 47 times is P'_47' = 1-P'_47 = .246.
So basically, the actual confidence value on that study is P=.246: there's about a 1 in 4 chance they would find a P=.006 result for one of the 47 food groups tested if the null hypothesis is true. The null hypothesis in this case being "diet doesn't affect telomere length".
The paper doesn't seem to list their hypothesis, but they say things like "all others were P>.05", which indicates that they were hypothesizing a P<.05 result. Doing the same math again for the probability of a P=.05 result occurring if you try 47 times, P'_47' = 1 - (1 - .05)^47 = 0.91025516807. That's over a 90% chance! In other words, by doing the test on 47 food groups, they nearly guaranteed that they would find a result within their definition of significance. This is very bad science, and a study with this bad of a statistical design never should have been funded in the first place.
This study has not been replicated as far as I know, but is often cited in pop science[3][4][5].
[1] https://academic.oup.com/ajcn/article/88/5/1405/4649028
[2] Note that this rule only applies to independent probabilities; the result of coin flip does not affect the result of the next coin flip, and the result of telomere correlation to one food group does not affect the result of telomere correlation to another food group. This rule won't work when applied to dependent probabilities, where the earlier tries affects later tries. For example if you draw an ace from a deck of cards (P=4/52) and don't place the ace back in the deck, the probability of drawing another ace is lower (P=3/51).
[3] https://www.livestrong.com/article/506649-foods-that-boost-t...
[4] https://resources.teloyears.com/diet-and-telomeres/processed...
It seems like we are wasting a lot of effort when any experiment is unpublished, when we could at the least be associating a data point with a particular hypothesis.
This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result.
The current approach also ties the experimental method to the result, requiring potentially fallible oversight of the experimental design. The hypothesis-testing aggregator reduces this reliance, which is perhaps more honest about our ability to design good experiments, would allow for a greater diversity of approaches, and could also be more efficient when evaluating results.
The most interesting aspect of this is if you could somehow create relationships between standardized hypotheses found on the hypothesis menu such that you could infer other hypotheses in a more systematic way.
You're absolutely right, but journals don't see it that way. Journals want to make money, and the articles which make the most money are the ones that prove an alternative hypothesis. This is one of the ways where for-profit publishing is harmful for science.
> This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result.
"Standardization of hypotheses" gets a bit tricky and I suspect the standardization process would be stifling. Part of the goal of replication is to improve on methodology, and part of that is finding ways in which your hypothesis wasn't clearly defined, or doesn't contribute to a larger theory. There needs to be some flexibility in which hypotheses scientists pursue.
A more organic way might be for journals to categorize articles as testing new hypotheses or attempting to reproduce old results, and strive for some ratio between the two (1:4 or somesuch).
No we can't. There are 2 (related) reasons for this. The first is that to say "after experiment X, we can conclude thing Y is happening with probability ..." you need to know something about how often thing Y happens on its own. This is also known as a prior. The other reason is that in this specific case, the prior for Y (that the coin is weighted) can be concisely summarized as P(Y)=0, because it's physically impossible to construct a coin that's biased to one side or the other (independent of what side it starts on). Some people think the coin flipping examples are bad because biased coins are impossible, but they're actually good for exactly this reason. We have a really good prior (from math/physics/geometry) that biased coins can't exist, so you're going to have to not only flip a lot of them to convince me otherwise, but also give me an explanation of how that's possible (i.e justify your prior)
Sally must be working using the wrong null hypothesis. If she attempts to reproduce Robert's results her null hypothesis should be whatever is the one he formulated. Perhaps that the probability of tails is at most 15%. Her p-value would be 2.66% (prob of seeing <= 2 heads) and if the world was fair, AJCFS would publish her study. But the world isn't and if Robert is a renowned scientist, significant+ is enough for him but failed replications of his work requires significant+++.
The most obvious criticism that stands out to me is that when a control group is possible, that's where your null hypothesis should come from. But in the contrived coin-flipping example, there isn't a natural control, and in practice, many times control groups aren't available. So I don't know if that criticism actually applies to this situation.
The next thought I have is that if your goal is to determine significance (which should be the goal), rather than to find a significant result (which should not be the goal), using the same alternative and null hypotheses makes the math for aggregating the results easier. The power of replication is that it's multiplicative: you can get an aggregate P by multiplying the P values of aggregate studies. This means a P=.2 study can become significant if it's replicated (P=.2.2=.04), while a P=.03125 study can become lose significance with just one failure to reproduce (aggregating Robert and Sally's results, P=.03125.3125^-1=.1). I guess swapping the null and alternative hypotheses is a viable strategy for bringing attention to failures to replicate and it's not mathematically invalid, but that seems like changing your hypothesis to fix a problem with the academic publisher, and it complicates aggregating the P-values of replication studies.
Finally, it feels weird to me to use Robert's hypothesis as your null hypothesis. When Robert performed his experiment, his hypothesis was a stab in the dark due to a lack of pre-existing data (since he ostensibly didn't know about the other coin flipping experiments). But when attempting to reproduce a result, you have data, so it makes more sense to base your null hypothesis off the results of the previous experiment, not off the hypothesis of the previous experiment. As such, since Robert had 5 heads, the hypothesis would be that the probability of getting tails on a single coin flip is P=0, which gives Sally a VERY significant result. The numbers are weird because the experiment is contrived, but I think the general concept holds.
Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Where is it written that our null and alternative hypothesis should conform to what is true, when our best ability to predict the future (as people) is based on our past experience? The world doesn't always behave predictably, and yet that seems to be baked into the assumption of every regression test ever run. Completely bonkers if you ask me.
I would probably keep the p-value, or something like it, for the rigorous analytical side, but I would make the assumptions to be tested much more rigorous. I would want to know not only why the test should be surprising, but I would want to know if our level of surprise is going up and down over time. It may be, for example, that as we accumulate more knowledge over time we are less and less likely to be surprised (or maybe more!) and therefore we should expect the p-values that elicit a surprise or not to change.
But overall, I think there is not enough emotional soul searching over what it means for a mathematician to be surprised. And since that is the primary axiom over which everything else follows, and a lot of statistical papers don't really think this through well, there is a ton of faulty analysis out there. Garbage in garbage out.
EDIT:
To give another clearer example - Suppose we want to test if a miracle drug makes people immortal, to give an obviously ridiculous premise. Then in this case maybe a p-value of .001 is most appropriate because it would overturn huge amounts of medical knowledge. In this case though we need a judgement call backed up by evidence, in the sense of evidence to convince another such as in a law trial, to make a convincing case that this is the "correct" p-value to use.
I think there is a desire in science to simply say, "Through sheer logic I have found the answer and therefore charisma/politics/judgement have no bearing on my analysis. And therefore I cannot be disputed on such things". This is an emotional viewpoint that seems in contradiction to the evidence, and it seems a common enough viewpoint that I don't believe it's purely a straw-man argument.
But then if the politics do exist how do we get around them? After all, if we start making broad based judgement calls on what p-values should exist then how do we make sure papers are not disputed for begging the question?
I would say that the best solution would be to force statisticians to get a third party to give them an appropriate p-value. Then the science has quickly, as many things, become a social problem - how do we organize such a system of statisticians giving each other appropriate p-values such that it is accurate but not corrupt? Unfortunately, this sort of thing is generally considered a "hard" problem.
I wrote an explanation for clinicians a while back that I people have had good luck with: http://madhadron.com/posts/2016-01-25-p_values_for_clinician...
> Likewise, we're supposed to assume that there is something magical about our prior assumptions?
No, we construct trials that a reasonable practitioner thinks satisfies the assumptions of the analysis used. And sometimes we learn that we had overlooked something and all those trials we did before we accounted for it were flawed.
Because that's how the math works. It isn't possible to compute a posterior without first having a prior, so you have to decide on a prior somehow.
You can dress it up and try to hide it, but you can't avoid Bayes' theorem forever.
There's decent working guidelines for statistical power, but a lot of the issues (especially around replication) I see are mainly due to sample sizes being too small to adequately detect effects, especially if those effects are small.
The basic idea is: Get more data, especially if you're measuring something subtle.
Though I would have personally used statistically instead of strictly in the sentence: >Strictly speaking, he says, “there’s no difference between a P value of 0.049 and a P value of 0.051.”
I 'm not sure if abandoning tests altogether is good though. What does it mean "it's detectable but not clear" for communication? How do you e.g. communicate global warming like that?
To get an idea how climate science is trying to communicate look at the summary for policymakers of the SR15 report: https://report.ipcc.ch/sr15/pdf/sr15_spm_final.pdf
They have different confidence levels they indicate instead of a single cutoff.
The problem with p-values is that the shape of the resulting cutoff just isn't what we're looking for. We want something along the lines of "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false?" is only an approximation at the best of times, and at worst, downright misleading. That's true even before we consider some of the other issues that my English kind of elides over, but the math contains; one of my problems with significance testing as it is commonly done is that there are actually ranges of hypotheses, and it really overprivileges "the" "null" hypothesis; I could write a decently-sized HN post just criticizing those two quoted words.
There is no rigid process that can produce the answer we really want, but I think we can provide a selection of better default tools. An example of one that has already been deployed to some extent is "power analysis", which is not a direct answer but lets people crafting studies analyze how big their studies will have to be before running them. We can build more tools like that.
The first is impossible to calculate by definition. The second can be derived from p value. Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used.
"Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used."
That's begging the question. The entire topic of conversation is whether or not the standards of justification are adequate.
There are some scientists thinking over this though. This is an idea from a neuroscience lab: https://www.researchmaps.org/
The idea was to create causal directed graphs for biology from the literature , which would be used identify what experiments are missing and thus inform future science.
The answer doesn't lie in hard-line stances for or against P values or with an alternative that will have its own set of problems. It lies with greater education of those who run experiments & those who consumer the literature about proper interpretation and other methods of analysis that should be used along side it.
It's not the statistical analysis that's the problem, it's the bad "science" and irresponsible journalism.
While their sources are supporting this statement, I'm getting mixed signals.
https://www.nature.com/magazine-assets/d41586-019-00857-9/da...
This is their primary source, of the statisticians calling to retire statistical significance. However, their primary reasoning is because statistics is misused to make erroneous conclusions. It seems like there is a lack of understanding about the philosophy and mathematics behind statistics that's the problem by its practitioners, not statistics itself.
Tightening significance threshold just makes this fishing more difficult, Bayesian reasoning doesn't help much either, because you have to guestimate reasonable priors. What does really help against fishing like this is requirement to preregister your studies.
What I predict would happen: - either lots of studies are allowed to preregister, most of which cannot reject the null hypothesis. You end up with a lot of "boring" null-result papers in those high-profile journals that nobody gets excited about and nobody gets promoted for and no media coverage happens, bad marketing for universities and research centers.
- or there would be a strict filter for the pre-registration, so that researchers cannot chase their gut intuitions, some authority would need to approve the study even before it's done. This hinders research and hinders the dissemination of truly unexpected discoveries.
The human incentives are way deeper than any one solution could touch on. The whole science funding structure requires flashy and sexy results that are just not possible to produce on that scale.
I see two things going on here.
Having additional 'oversight' for the scientific validity of a planned experiment, is presumably a good thing.
I don't see that it would be necessary to set out to prevent researchers investigating their intuitions.
The purpose of a paper is supposed to be to contribute something new to a field, not just perform a statistical study.
But you or I cant say that now as so many organisations use "studys" as the basis for their field while maintaining that they are scientific.
For me, the fact this is even being discussed really is good news.
The physics community is pretty good about dealing with both of these shortcomings. You often need a lower p-value of around p=0.003 (3 sigma) or lower to say something was significant, and a few more sigmas to claim an actual discovery. And in addition to this, you're expected to include a trials factor to correct for multiple hypotheses. It's not perfect, but it makes claims of statistical significance more meaningful.
I am very happy to see it in Nature.
One of the comments below references this XKCD comic [0], which IIUC is an example of p-hacking.
But in that comic, the only difference I notice between the original hypothesis (jelly beans cause acne) and the p-hacked hypothesis (green jelly beans cause acne) is whether or not the hypothesis occurred to the researcher at the beginning of the study. And I don't understand why that would bear on the importance of each hypothesis.
The reason you need to pick your hypothesis ahead of time is because otherwise you may use intuition or just your eyeballs to find relationships in the data that are only there due to random chance. This will supposedly increase the probability of detecting spurious relationships and erroneously rejecting the null.
IIUC, you're pointing out that when examining the empirical data gathered during an experiment, it's often possible to find some identifiable subset of the data that are consistent with a refined version of the original hypothesis. E.g., maybe jellybeans in general don't correlate with acne, but green ones do.
Assuming that the experiment is part of a larger effort to build or refine some model, I don't see the problem.
Suppose that someone refrained from p-hacking during the first run of that experiment. IIUC, they'd look at the experimental results, and wonder if they had missed some "X" factor. So they might conject that jellybean color was relevant, and rerun the experiment with the hypothesis, "consumption of jellybeans, but only of a particular color, correlates with acne." And (assuming sample sizes were big enough), the data gathered during that second experiment would likely confirm that green jellybeans correlate with acne.
But what's the point of having run that second experiment, when they could have just reached the same conclusion by testing additional hypotheses from the first experiment's data?
It seems like regardless of whether you just ran one experiment and did "p-hacking", or instead ran a follow-on experiment, you end up with the same refinements to the model you're working on.
I think you overestimate the intelligence of people prone to misunderstanding. I don't think they're likely to understand big words like "statistically" or "discriminable" when they already don't understand big words like "statistical" or "significance".
They do understand what "significant" means, they use the world every day. The problem is that it means something different than the intuitive every day meaning when used in context of p-values. Using a word like "discriminable" might help clear things up since it's a word that doesn't have have so much meaning packed into it already.
It's like when a mathematician says that something is "almost always" true, they mean something very different than when a non-mathematician says something is almost always true.
Yes that's my point. Using words that are hard to understand or only understandable in the context of P-values won't solve the problem; the problem being that the general public won't understand the subtle differences in meaning
Frequentism is antiquated.
> Is there a better way to judge if a study is solid?
> Unfortunately, there is no single alternative that everyone agrees would be better for all experiments.
nobody has solved the trust issue
Not as "This research has now found X and is in a reputable journal, so this is the new state of science from now on".
The problem is, academics are pushed to writing more and more papers. There's no time to discuss back and forth too much about an already published paper, there are new papers to write! The goal of the project was to publish a paper, published=won, end of story, next project on.