It will be much harder to call findings ‘significant’ if a team gets its way
sciencemag.org
sciencemag.org
I think in general though, at least in biology, people are waking up to the fact that p-values aren't magical, and that having a really small p-value isn't a goal in and of itself. It is, however, necessary to do some statistics on your data to get it published, but the p-value is just checking a box more than being used as a tool for discovery.
Yesterday a cool dataset was released from Jeff Leek which has over 3.6 million p-values from scientific literature[1]. The distribution is fun to look at by discipline[2].
[0] http://rpsychologist.com/d3/NHST/
Sometimes, it is necessary to use a significance test in order to convince the reader that your conclusion based on your interpretation of your observations is sensical, which would be "checking a box".
Other times, and as a frequent reviewer I do see that, infrequently, but still, where the authors see something where the only thing to see seems completely anecdotal (small N, large or fake error bars, no significance or ranking test), or to support their innacurate interpretation of a phenomenon (I see that with young and inexperienced graduate student in a young and inexperienced PI's lab). In these cases, having a proper experimental setup, sample size, and analysis, is part of the discovery.
At the end of the day, it is important to make sure that science stays evidence-based and statistic is one of the tools that can assist, but only assist, in doing that.
And here it is zoomed in: https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg
These figures look very damning to p-values. This basically looks like they are made up anyway. I haven't checked the methods though so who knows.
Edit:
Well there are 33 "pvalues" > 1 and 20820 that equal zero, so this dataset probably could use more cleaning.
This is a better pic: https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large
Can't really see any difference in fields, besides some artifact in plant biology. Which makes sense since it's just a statistical test.
require(tidypvals)
allp = allp[!allp$pvalue > 1,]
allp_eq = allp[allp$operator == "equals", ]
hist(allp_eq$pvalue, xlim = c(0, .15), breaks = 1000)
The overall function is a p-curve, here's a paper on it: http://journals.sagepub.com/doi/pdf/10.1177/1745691614553988
Be careful with this. Larger sample sizes are more likely to give you a significant result even if one doesn't exist.
Say my null hypothesis is that X=100. The alternative hypothesis is X>100.
What if in reality X is really 100.5? Depending on the problem domain, this may well be the same as the null hypothesis. But a larger sample size is much more likely to give a significant result.
There are ways to fix this, but one should just be aware, though.
> What if in reality X is really 100.5?
Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.)
> Depending on the problem domain, this may well be the same as the null hypothesis.
No, X = 100.5 is not the same as X = 100, irrespective of problem domain, so long as the mathematical symbols have their usual definitions.
> But a larger sample size is much more likely to give a significant result.
Yes, if something is only just barely true, it is more likely to take a large sample size to distinguish it from the case where it is false. But since in this case, the alternative hypothesis actually is true, it's not a problem that a bigger sample is more likely to reject the null hypothesis.
Mathematically, yes. Practically, not always.
In many real world effects, we're not 100% accurate on what the null hypothesis is. We may think the existing process suggests a the temperature increase of 1C, and so we set that as the null hypothesis.
But in that problem domain, we may have difficulty being more precise. Suppose that 1C in the existing process came from the mean of measuring a sample of 100, so the current belief is that the expected value of the increase should be 1C. Suppose in reality it is 1.02C. Now we're proposing a new process and claim it is better than the existing one. This time we take a much larger sample, and see the number hovering at 1.02C - we can then claim the new process is better than the existing process and have a p value to demonstrate it.
The math will be correct, and the conclusion will be wrong.
P values measure statistic significance, not practical significance. If you are extrapolating practical significance from statistical significance, then you have a problem that is unrelated to sample size.
If you never extrapolate practical significance from statistical significance, then you have a problem. p values exist for the purpose of doing that. The problem in in research isn't the use of p values per se, but the misuse and misunderstanding of them.
An argument to do away with p-values demonstrates a basic misunderstanding of them.
The null hypothesis is the simple negation of the affirmative hypothesis; anything else and you can't use p-values at all.
If you know what the hypothesis you are testing is (and if you don't, you have a fundamental problem), you also know what the null hypothesis is.
The proper affirmative and null hypotheses in the test you propose are “process 1 has a greater measure on the dimension of interest than process 0” and “process 1 has no better measure on the dimension of interest than process 0”. Normal statistics on parallel measures of both processes will give you results which may or may not allow you to reject the null hypothesis at a given p-values, but (except that the smallest p-values you can get goes up with sample size, so that you can't, rightly or wrongly, reject the null hypothesis to certain confidence with too small a sample size) the probability of rejecting the null hypothesis incorrectly does not go up with sample size.
If you do the statistical significance test correctly, you won't be able to show the new process better by taking lots of samples for it, since the variance of the old measurements dominates the uncertainty. (I think, haven't done the math to show it.)
If you get wrong conclusions using correct math, your assumptions are wrong. In the case of your example, the incorrect assumption is treating an uncertain measurement as exact when comparing it to another uncertain measurement.
>If you do the statistical significance test correctly, you won't be able to show the new process better by taking lots of samples for it, since the variance of the old measurements dominates the uncertainty. (I think, haven't done the math to show it.)
Higher sample size means lower variance:
That's for a single population. If you are trying to compare two populations, you have two empirical means and two variances. When you do a t-test [1] to compare your measurements, the test statistic has a variance that is approximately computed as
variance = variance_1/sample_size_1 + variance_2/sample_size_2
Adding more samples for only one of the populations means that this half of the sum vanishes, but the other half remains unchanged, thus guaranteeing that the variance never drops below a certain threshold.
[1] https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_un...
Your statement didn't address anything in my comment.
"It's confounding with the meta-analysis the data comes from: Brodeur 2016 is the only one that did econ; didn't collect <= or >= from papers"
I can't even begin to imagine how one can collect that number of papers in a machine-readable format without ending up like Aaron Swartz.
It's worth noting the datasets (except Brodeur for economics) include "less/greater than", e.g. p < .05
This is the reason for the bimodality, and also for economics missing the second peak
I'd suggest that a better, broader, and more practical solution is to provide a public clearing house for studies and associated data -- let's assume it's a website.
a) When you start a study you must describe your study, methodology, and hypothesis, and publish these things on the site. It should be possible to find the study the same way you find any paper that results from it.
b) All data must be stored in the same place (but not made public) as it becomes available, and this must be demonstrable. This data should be available to anyone trying to replicate, review, etc.
c) Final data sets and analyses must also be also be available, ideally with the code used to do the analyses.
All of this should be prerequisite for review and publication, but also for anyone working in the field should no papers result.
This solves a lot of problems, including making it easier to conduct meta-analyses when looking for effects that aren't significant in smaller studies, and also address the lack of availability of negative results, and even help combat theft of methodology (e.g. where someone sees work in progress and is able to publish using the methodology). It would make meta-analysis easier, and allow researchers to better take advantage of existing, unpublished results (e.g. to hone their study methods, etc.)
The first, theorists, design interesting studies with open ended questions, explain how the study would provide useful scientific results, or theorize about the implications of existing published results.
The second, practitioners, are hired to actually run the studies according to the specifications provided. The practitioners would publish null results of course. Practitioners would also have to acknowledge how many of their results are significant, so it'd be transparent if they were just confirming everything.
A bit like barristers and solicitors in many common law countries, a distinction that isn't strictly necessary for the system to work, but prevents some conflicts of interest.
It's another way of saying "First, take a spherical cow..."
A rigid rule of pre-registering "study, methodology, and hypothesis" would have prevented me from even launching the project. At the very least, I would have had to register a formal study after my results were already established to a decent degree of confidence. It would have been wasteful and redundant, not to mention risky, since anything can happen to a grad student to prevent them from finishing a degree (illness, loss of faculty advisor, etc). At best, time is money for a grad student who wants to get out of school ASAP and start earning a salary.
My project produced a humble but solid result, and launched a new research program for my advisor, that bore fruit for another 25 years.
Now, I wasn't doing a "study," as in a life science study looking for a statistical correlation. I reported no p-value. So maybe it doesn't apply to all kinds of scientific work. And I appreciate the need to restore come kind of integrity to the medical and behavioral sciences in particular. I don't know how to help those fields, and maybe pre-registration only applies to them. But I'm concerned that declaring rigid rules for science across the board will have unintended consequences.
Edit: Another issue is a student getting "scooped," where some other lab reads the plans and rushes the work to completion. There are labs in chemistry and biology that are notorious for doing this. If a student doesn't at least have an informal guarantee of priority over their own work, they won't take the risk of launching a project in the first place.
If you get null results, you have to report that.
assume that, to simplify, that every research paper out of there has findings with p-value < 0.05 - that means one in 20 papers has wrong conclusions in it.
or that one in 20 teams will not be able to reproduce findings.
0.05 is just removed enough from coincidence that, given a solid theory, corroborates the findings. but running across data to find correlation and using 0.05 as threshold is a surefire way to find something that has no actual significance beyond mathematics and zero ground in reality.
> [Bates] called the proposal “a risky distraction”... [It] wouldn’t address many other practices linked to irreproducible results: poor study design, a bias toward publishing positive results, and the practice of “p-value hacking”—fishing for significant-looking results from a huge number of hypotheses.
Since the reproducibilty crisis started getting attention, I've noticed that these problems are really ubiquitous. Tweaking study designs and modifying hypotheses after collecting data are very commonplace and much more subtly dangerous than a potential lack of statistical power. Gelman says "Valid p-values cannot be drawn without knowing, not just what was done with the existing data, but what the choices in data coding, exclusion,and analysis would have been, had the data been different." [1]
This proposal also says nothing about effect sizes, which also should play a role in how we interpret scientific results. We should probably draw different conclusions given a tiny effect at p=0.05 versus a large, obvious effect at p=0.05.
[1]http://www.stat.columbia.edu/~gelman/research/published/asa_...
I'm not sure that "large, obvious effect" and "p=0.05" are compatible, though.
> a bias toward publishing positive results ... fishing for significant-looking results from a huge number of hypotheses
If we accept a lower p-value as significant, there will be fewer hypotheses that have positive results, and therefore less to publish.
Should we really do nothing at all until we can come up with a perfect solution to all aspects of the problem? This won't fix the real problem, but it will reduce the number of pop-sci articles spouting bullshit results that don't reproduce next week.
I agree that doing something is better than doing nothing, but I disagree that this will reduce bad results in a meaningful way. Setting a more stringent threshold for significance will elimnate studies that reach the wrong conclusion due to a statistical fluke, i.e. there's no effect but due to random chance, the noise in the data makes it look like there is. But IMO these are relatively rare. It seems to me that the vast majority of bad studies reach significance by virtue of methodological issues that enable one to reach any arbitrary level of significance just by trying hard enough. If Daryl Bem continued doing what he was doing, he would have eventually reached p<0.005.
edit: I just glanced over Bem's paper and he reports several p-values under 0.005 and several other very close to it.
It would be far easier, and more honest, to simply report the P-value obtained by performing a statistical test without adding the term "significant".
Essentially the more likely your hypothesis, the higher your P value can be, and conversely unlikely hypotheses require stronger evidence.
I don't think this solves the problem but maybe it makes that assumption about the hypothesis likelihood more specific.
Let us propose that correction for multiple comparisons be a life long endeavor. At the start of a scientist's career as principle investigator, p<.10 is fine. But by their career's 10th comparison, they have to meet at Bonferroni p<.01, and at their career's 100th comparison, now they have to meet at Bonferroni p<.001. When they are old and gray, or doing genomics, at their 1000th comparison they have to meet at p<0.0001.
Interesting dynamic.
1) Not really addressing the issues behind the replication crisis
2) Doing the exact opposite of what should be done (raise the significance level)
Undue emphasis on p-values is bad, but what you're suggesting is bonkers.
Look, not every paper is worth publishing. There are many ways to write noise that can pass for signal given the reader's level of understanding (e.g. crack pottery and "fashionable nonsense"). The usage of p-values implements self-filtering to some degree, and makes everyone better off by increasing the likelihood that at least some papers aren't complete nonsense. And that's good.
The only issue is that it is a much weaker constraint on publishing than it was previously thought.
Many more results should be documented than currently are, though, including negative results.
But I agree that changing p-value thresholds is unrelated to that.
Also, in the usual case where the null hypothesis is "no difference", no one really cares about what the p-value is telling them anyway. They care "is my theory accurate/true", when their theory predicts something other than the null hypothesis.
I figure we may as well set the significance level in these cases so that everyone "wins", and go on to use analytic techniques that actually address the question of interest.
>10 significant things that WILL shock you, #7 is this team
>THIS crazy trick that will make significant much harder [teams hate him!!]
>This finding prank gone significant [gone sexual!?]
Sorry, it was such a click-baity title for no reason.
Any golden value for p is bad. If I go back to my statistics text book (and I assume all of them say something to this effect): The appropriate value of p depends on the particular topic of study.
Not on the whole discipline.
And definitely not across all studies and across all disciplines.
You cannot pick a value of p and say "this is good". Pick too low and you'll likely not be able to reproduce many legitimate results.
Scientists would still be free to make use of hypothesis testing to guide their research. In publications, one could use Bayesian methods and report a full posterior. One could also just give enough information about the null hypothesis and whatever distributions are involved to allow the computation of a p-value.
However, any attempt to explicitly publish the p-value itself would result in creative and embarrassing punishments.
I fail to see how this is superior.
I'd never advocate banning a legitimate tool just because of abuse.
There are a lot of researchers who have a limited knowledge of statistics and who basically treat the p-value as a magic number.
I don't have any sort of philosophical problem with p-values or the tools of hypothesis testing, but I think it would be good for science if the magic number were less salient, if that makes sense.
Eg. The Journal of Wacky Speculation publishes "The Moon is Made of Green Cheese". Using a highly sophisticated cheese detector, the authors measure results consistent with this hypothesis. They calculate it would happen only 4% of the time if the moon were not in fact made of green cheese, and therefore the result is significant. The cheese detector works perfectly, the calculations were accurate, but this does not mean the moon is made of green cheese! We've been there, and we're very certain that it's made of rock. Because the prior probability of the hypothesis was so low, the "significant" result was most likely a fluke. The Journal of Wacky Speculation specializes in this type of hypothesis, so they should use a much lower threshold for p.
Meanwhile, the Sensible and Boring Society publishes a study showing that round wheels roll better on ordinary roads than square wheels. We already had good reason to believe this, so p<0.05 shows it's most likely true. The Sensible and Boring Society publishes tests of hypotheses with higher prior probably, so they can use a higher p threshold.