What a debate about p-values shows about science
vox.com
vox.com
> Yes, you’d want a very low p-value in a study that claims mental telepathy is possible. But do you need such stringent criteria for a well-worn idea? The high standards could impede young PhDs with low budgets.
Watching someone try to be Bayesian without using Bayesian techniques is painful.
> There are also new, advanced statistical techniques — like Bayesian analysis — that, in some ways, more directly evaluate a study’s outcome.
Almost as painful as watching Bayesian analysis described as "new."
Less snarky: not really
Also Bayesian analysis (as a mainstream stats technique) is relatively new as it has only become mainstream with the advent of cheap computing power.
Publish negative results.
I'm kinda embarrassed by my negative results, such as they are, but I'm clear in this latest case that I wanted the method reviewed most, since others then may be able to use it.
(Warning: not a real scientist, but being validated by real scientists in the case in question.)
There are times when negative results pose new problems or offer new questions or methods. Those should be published but don't normally get you much praise. A blanket statement about publishing negative results is just going to get overwhelming information that nobody has time or interest in.
Frankly, there are more productive things that occupy a researcher's time. If you want to know what a researcher has tried and failed, then chances are you're a professional in the domain. You should already know how to contact them and you can ask them to share their journals and logs with you.
But, nobody is going to submit all their negative results. Nobody is going to index, catalogue, cross-reference, fund, and disseminate these massive negative results.
If you want to know all the things I've tried and failed (I am a retired scientist), it would take multiple volumes and far more time than I'm willng to devote. Prior to retirement, I had positive results to publish, more things to try, and to find a way to make it commercially viable.
Those acts are far more valuable than publishing things that got a negative result. Publishing those would take all my time and I'd never have gotten anything done.
The system works. The system was designed as it was for a reason. Yes, some negative results should be published but they are exceedingly rare. The problem is a lack of effort to reproduce prior to acceptance and low-quality peer review. The solution isn't to publish more stuff, it's to publish less and to ensure the quality is there. The solution probably should also include open publishing. Instead of locking it up in expensive journals, make it more open so that more people, even layman, have the chance to review and replicate it.
Speaking of which, I mention again that I lament the lack of emphasis, appreciation, and support for the citizen scientist. I'm a scientist. I'm nothing special. I'm wrong more times, by 09:00, than most people will be all day. Stop putting scientists on a pedestal or in ivory towers.
You really don't want our many, many volumes of negative results. Just because it's got a Ph.D. tacked onto the end of the author's name doesn't mean it is valuable.
Really, just uphold the standards that many seem to have forgotten. You're correct in that some negative results should be published, but they are few and far between. If anything, we have too much being published and a distinct lack of quality.
I will add this...
Journals only want a trimmed down paper. A journal may want you to trim your paper down to 10 pages, some want even shorter. Your research may be 1,000 to 2,000 pages but only a tiny bit gets published, or even submitted. That full research will have many of the negative results, as may the 'private' journal that the researcher worked from. If you want that information, you can probably get it emailed to you. The institution may have it in their archives, if the researcher is no longer available.
So, negative results are out there, but most don't really need to be published. Most researchers don't have time, resources, or inclination to publish negative results with dubious value. Sometimes a negative result is potentially valuable, as it introduces something novel. Those get published with some frequency, though rating it as novel and anticipating its ability to further the arts is easier said than done. Even still, those rarities are even more rarely deserving of accolades. There are no participation trophies in science. That sort of thinking is kind of how we got into this mess. Somewhere along the lines, someone should have said, "No, that's garbage. Don't publish it under the heading of science or with the prestige of this journal/institution attached."
Science doesn't care that someone tried hard. Science isn't about applauding effort but is about results. Negative results with significance may, very seldom, be worthy of publishing.
Publishing the vast majority of negative results would be a waste of time, funding, and resources. Maybe you're unaware of the many failed experiments across the domains, but they are plentiful.
If you want to read them, the data is still collected - it's just not published. There's no good reason for it to be published in all but the rarest of cases. Just email the researcher, it will be in their journal. If they published, it's in the stack of notes and was trimmed out because it takes space, time, and money while having negligible value.
It's great that someone put a lot of effort in, science does not care. Frankly, we are publishing too much junk as it is. Adding to the list of junk isn't going to solve any problems, it's just going to create problems.
The solution is to increase the quality if what is being published, not to increase the noise to appease some emotions. Again, there is no participation trophy - nor should there be.
By declaring something as "too dumb to study", you're a priori declaring the result, which is bad science.
Interestingly, underpowered positive results are likely to be published, but unreproducible. That is, the ratio of false positives will be higher among underpowered studies. So, the literature of underpowered studies will be enriched for false positives.
So, underpowered results, whether positive or negative, are less interesting.
The idea is probably only applicable to a small subset of scientific work (not good for exploratory and very novel stuff, for instance), but in certain specific areas it might make publication of negative results a more viable path for scientists because they don't have to fear that their idea was bad or they seem incompetent because of a negative result. (Since their design was accepted, at least some other people thought it was a good idea too.)
Of course, based on the current p-hacking, publish at all costs environment, it's not clear you need to be particularly useful to be published. I like the idea that has been promoted of requiring a certain number of novel replications in a tenure application as one way forward... at least as a goal or professional shaming tool.
That's just the externalities around publishing negative results. The major reason being that it will tell people where the minefields are, resulting in people not all reproducing the same negative results.
What holds us back, it seems to me, isn't process, it's culture and inertia.
There is great value in publishing negative results in the form "I hypothesized X, did Y, and turned out X was wrong". It's my experience these negative results are often well controlled and trustworthy as there are few things that make one trust a method as much as holding onto it for dear life because you have spent so much time building it. Incidentally that's not always as true for positive results where the incentives are tilted towards stopping once you get ahead.
It may be true that positive results are more useful than negative ones, but that doesn't mean we should put zero value in negative results. I think that's what parent was trying to point out.
(Warning: In Scientist-Engineer limbo)
Or just don't publish the results at all, only publish the methodology and make the results available elsewhere along with the data and source code.
It's rare, but sometimes people without any degrees get published in reputable journals.
If I were in your shoes, I'd do what I can to get published in something reputable and buy a bunch of printed copies. I'd then autograph them and hand them out like a rock star.
When I was first published in a journal, I brought a copy of the journal and a couple of 'official' single article prints for my mother. One of them she hung on the fridge and even stuck a gold star sticker on it.
Of course, buying a copy of the whole journal and single article reprints was a lot less expensive then. I'd still get physical copies, just to show them off.
Best of luck and feel free to yell, both when you've got it published and if you have any questions about formatting or want a proofreader. If it encourages you to publish, and encourages people like you, I'll do it. I really do wish there were more citizen scientists, or at least people who lack the standard credentials, submitting their ideas, being interested, and being willing to share their insights and research.
Either way, best of luck! You've made my night. I love stories like that.
uninvolved@outlook.com (generic public email address)
I do have my name on a pukka scientific paper from many moons ago, as the summer help for a PhD, where the real authors slung me in the author list out of kindness.
But I do like the idea of a gold star. Then I can be 'award winning'! B^>
That was molecular biology, this topic is home heating efficiency. That one is online for all to see, this one is too but is a bunch of very boring docs and part of a GitHub repo!
> The researcher basically asks: How ridiculous would it be to believe the null hypothesis is the true answer, given the results we’re seeing?
but I think they (indirectly) correct it pretty well in the next section.
It isn't published yet, but Gelman and co's recent piece on this seems like an important contribution to the debate [1]. Short version: take the p-value down from it's pedestal, abandon the dichotomous view of is/isn't significant, and consider it with all the other evidence and data.
[1]: http://andrewgelman.com/2017/09/26/abandon-statistical-signi...
> The case against p<.005
> The real problem isn’t with statistical significance; it’s with the culture of science
Right. The substance happens in the last 10% of the article.
...and there isn't any solution or recommendation.
Unfortunately, I think this article is really long way of saying, "There exists p-hacking". I was hoping for more :(
https://fivethirtyeight.com/features/science-isnt-broken/#pa...
Personally I much prefer ROC curves, they are much better at showing the difference between distributions. Still, nothing beats the raw data and healthy skepticism.
That's a common misunderstanding of the p-value, but that's not what it means. (It means: assuming there is no difference, the probability that we'd nevertheless see a difference as big as we are seeing or even bigger is less than 5%).
My motivation to co-write was strongly influenced by being called out for p-jacking. Its remarkably easy to fall from grace. I did. walking back is very hard, when peer review doesn't want to have a polemical discussion.
A robustly designed experiment should produce repeatable, intuitive results. Anyone looking at the presented data should be able to reason the result without needing to conduct a statistical test.
Obviously, these tests are important, but people focus too much on the p-value rather than designing an experiment which will produce a meaningful result (regardless of whether the result is positive or negative).
Proper experimental design is like a cake, and p-values are the frosting. Everybody wants cake, most often with frosting. Nobody wants frosting without the cake, or at least those that do have questionable taste.
And true, there are certain kinds of experiments that can only be done statistically. However, depending on the field, it can just be easier to do, and really should just be considered lazy science - again the details matter.
In my field (biochemistry/genetics) there really is a very real effort to design experiments that do not require statistics to verify the result. And that effort in design is rewarded amply when the result of the experiment becomes quite clearly binary in nature, from multiple angles, rather that a statistical "probably".
However, if you have an experiment that pulls X from the system, watches Y fail, then readds X in a new way to recreate the system, you can be much more certain that X influences Y. That becomes a much desired, "elegant" experiment.
The later experiment is often harder, and might not even be proposable until you've seen some statistical effect. However the scientific results from that experiment really don't hinge on statistical signifigances. The results are either binary in result, or better yet, contradictory - elucidating hitherto unknown variables (something a purely statistical result won't help with). And if mechanism is shown in multiple systems, with multiple techniques, the results can very quickly become near-definitive.
Yeah, in molecular biology this always seemed to be the most convincing evidence. Deleting a gene and then adding it back. Sometimes going a step further and overexpressing the gene. You can generally publish that data without any statistical analysis.
> The later experiment is often harder, and might not even be proposable until you've seen some statistical effect.
I think the better journals/reviewers can push back at this step, though: If you don't have a demonstrable mechanism, you can't publish. Don't let people stop at "...this gene is important (p < 0.005)." (I know not all things are as simple as gene deletion and complementation)
A trivial case would be figuring out how fire works. You have have heat, fuel, and oxygen. Take any of those away, and there is no fire.
This can be demonstrated by direct experiment. But it’s unfortunately a pretty blunt tool. Because the real (and non-trivial) case is that this isn’t strictly true.
What you need is not specifically oxygen, but rather anything that can function as an oxidizing agent.
Still demonstrable by direct experiment, but more difficult. And how you want to understand fire is directly related to what you want to do with it: create it? Or make it go away?
If you want to make fire, it might be enough to understand the basic three element. You stumble onto the fact that magnesium will burn, and that’s good enough that you don’t look any further.
If you’re interested in putting fires out, it’s also “easy” to show that sumberging a burning object in water puts out the fire by direct experiment. Until you try to put out a magnesium fire like that. Similarly a phone battery that’s burning up will get worse if you dunk it in water.
Then on top of that, there are things that act like fire but are fundamentally different. Nuclear reaction as opposed to chemical ones.
Now, of course, we have a pretty good idea of how all of this works at a molecular or atomic level, but direct experimentation gets us a certain way down the path of investigating what’s really going on, even though it’s pretty imperfect.
But pretend that you have no knowledge of chemistry or physics, and you goal is to understand what causes fire. What you do know is some math and some statistics. How do you design a statistical approach to figuring out what makes fire? Or what makes fire go away?
You know that fire is related to heat because it’s hot. But you don’t know how. We also know that fire is related to wood because that our most frequent experience with fire. We think fire causes heat because everything that’s on fire is hot, and only some things not on fire are hot.
So there’s our null hypothesis: fire causes heat. The obvious alternative hypothesis is that heat causes fire.
And because we’re somewhat primitive in this example, we are really focusing on wood.
So we sample as many different types of burning wood as we can find. We go through the effort of finding samples of 1800 different types of tree on fire. (Doesn’t matter how the fire started, it just happened).
We find in every case that the burning wood was associated with heat of a certain temperature. So we accept the null hypothesis and reject the alternative.
Obviously, there’s a shit-ton of things that are obviously wrong with that statistical approach. And it probably seems comically bad to everyone. But the fact is that once you wade through the jargon specific to a given field, that’s really what you’re dealing with.
Null and alternative hypotheses that are causally confused, accepting rather than failing to reject . . . The whole thing. It happens mostly in fields where you cannot discover causal mechanisms through greater understanding of the platform. Social sciences, Econ, neuroscience, Climatology, etc.
And in the bad case above, note that the p-value would make no difference. The design itself and the misinterpretation is to blame. Not the p-value itself.
Yes, I think p-values and hacking them is a huge problem. But not because people have a habit of creatively interpreting data to get published. It’s because a focus on p-values encourages bad bad experimental designs, and reviewers don’t pay enough attention to that.
In any case, while I can't say I agree (and you gave the reason why), it sure is a more _exiting_ experiment if the result is immediately apparent!
(Gaither's Dictionary of Scientific Quotations gives the attribution "In N.T. Bailey The Mathematical Approach to Biology and Medicine")
I need this tattoo'd on my forehead.
Part of the problem, though, is that scientific results are often counterintuitive, or (particularly in the social sciences) are easy to view as obvious after we know what the answer is.
One famous example: the sociologist Paul Lazarsfeld once gave a talk where he asked whether urban or rural men had adjusted more easily to the day-to-day routines of military service in World War II. There are equally obvious reasons why rural men would adjust more easily (e.g. they have more experience with physical labor, have experience with guns) or urban men would (e.g. they're used to crowded conditions, have more experience with a hierarchical division of labor).
The only reason they are counterintuitive is because the current model of intuition is flawed.
A good experiment creates new intuition by creating a new model by which once counterintuitive results become intuitive.
This is an appeal to complexity fallacy.
A result may initially be counterintuitive, but it is the the duty of the experimenter to alter their model to fit the new data. To explain the results in terms of known phenomenon and develop a new intuition which includes the unexpected results in the model.
If you are having trouble fitting a result into your current intuition, it is because your intuition is an using outdated or improper model of the phenomenon.
Nobody is born with intuition. It is learned, via bayesian style updates of information to model reality. The more effort we put into quality updates of our intuitive model, the better our intuition of reality becomes.
P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise".
Then researchers should always put more emphasis on confidence intervals. Readers of papers can then see if the size is enough to make the results relevant and not likely caused by experimental accident. Plus given the perverse incentives researchers are subject to, maybe assume that the real effect size is probably closer to the lower bound of the interval.
Null hypotheses have limited usefulness even at p=0.005. A tiny systematic bias in the experiment can make it cross that threshold and there are _always_ at least small biases. These can be caused by not exactly calibrated instruments or small differences in the way tests are performed by different researchers on the team etc. Inherent in null hypothesis testing is this ridiculous assumption that there are no systematic biases. This is its fatal flaw.
The null hypothesis test never tells you that the effect is zero or that it is any other value, it only gives you an hint about whether or not the data is too noisy to say anything at all.
It's much closer to, this result rises above the noise floor. Suggesting but not implying a possible link between X and Y, fell free to investigate.
No, that's not what P=0.05 means. This misconception is exactly why p values are so problematic. The question we want to answer is unanswerable, so instead we subtly substitute a different question in its place and don't even realize we've swapped concepts.
There is absolutely nothing valid about your "flip" (called transposing the conditional) and the damage done to the human species by people doing the "flip" is too huge for most to comprehend. Sorry, but I really cannot overstate the problems with this "flip".
Perhaps people who get paid by grants should get three chances regarding this flip. If they strike out they are banned from ever being funded by NIH, etc for life. The result (either 99+% of current researchers would be culled, or p-values would become appropriately rare) would do more to advance science than any other idea I have seen.
Sure there are debates about priors. These are unavoidable. The hidden assumption in my example is a 50/50 prior on too noisy, not too noisy hypotheses. IMO a reasonable uniform (maximum entropy) prior to start a discussion from. Of course, one should be more precise when doing this type of analysis in a paper. The reason a good prior is hard to define here is not because you shouldn't do the flip, it is because the question that the t-test is trying to answer doesn't make much sense.
How can saying false things make your science "about the the real world"? The only thing I can think of is that by "real world" you are referring to using the publication of massive amounts of incorrect/questionable information as a metric for success. Yes, that is indeed the sham going on.
>"There is a reason humans naturally and instinctively do it when they talk about experiments. This is how you describe things in real world human language."
I have no problem talking about data without transposing the conditional, if you do that is some issue with your training.
>"Sure there are debates about priors."
The error you advocate (of transposing the conditional) has nothing to do with priors.
Also P(A|B) = P(B|A) if the prior probabilities are equal, often a reasonable prior.
Ok, but I fail to see the relevance of this to anything.
>"Also P(A|B) = P(B|A) if the prior probabilities are equal, often a reasonable prior."
Here, A = Hypothesis and B = data (or vice versa). You are claiming that the "probability of seeing your data whether or not the hypothesis is correct", is often near the "probability the hypothesis is correct independent of your data"?
Based on what? What principle leads you to think these two probabilities should be near each other? BTW, if you actually have one this is like the holy grail of statistics.
Shouldn't that be "the probability of seeing the data, _given_ that the hypothesis is correct" and "the probability of the hypothesis being correct, _given_ that the data has been observed"?
The claim is that it is reasonable to assume P(A) ~ P(B), so that P(A|B) ~ P(B|A). I am referring to P(A) and P(B) in the equation.
I agree there are debates surrounding priors, especially when you get into higher order, meta or purely hypothetical territory (there is no actual noise generator here) like this, but I think these are unavoidable.
For this to be true, P(observations) ~ P(pure noise generator) as you said earlier. Anyway, you can look at it from either perspective, doesn't really matter.
On what basis do you claim this is reasonable to expect? I suspect you have none whatsoever... in fact simple thought experiments will lead us to the opposite conclusion, and there is plenty of data telling us the same.
This is incorrect. A p-value is the probability of observing a value of the test statistic that is at least as extreme as the reported value, assuming the null hypothesis is true.
Now, the exact values of a given data set are vanishingly unlikely, so we actually ask about the probability we'd say this data or even more extreme data given that we're just looking at noise. That's a typical p value.
The whole terminology with values being "extreme" is just unintuitive and unwieldy, IMO.
If I wanted to see if native Hawaiians are citizens and didn't have access to law, real data, anything like that, I'd sample from the population of Hawaiians, not citizens as a whole, and then regardless of the null hypothesis being "are citizens" or "aren't citizens" I think the results would be overwhelmingly one-sided enough to work out. And the South Africa example is asking about 1 single person? That not a question of science, then, just a question?
1. You have a null hypothesis you assume to be true
2. Based on this null hypothesis and the central limit theorem, if you theoretically conduct many experiments you expect some summary statistic to have a specific distribution around the 'true' population summary statistic and 'true' population variance, which you don't have so you assume it to be the null hypothesis
3. You compare your experiment's summary statistic to the hypothesized distribution. If the probability of seeing your summary statistic is below some threshold, you say it's unlikely to see this summary statistic again.
But the probability of seeing that summary statistic again actually depends on P(H), which NHST doesn't provide any information on. My examples were meant to highlight the nature of conditional probabilities, rather than how real life experiments are conducted.
"if I try to decide that native Hawaiians are US citizens, and the null hypothesis is that they are, but since only ~0.2% of total US population is native Hawaiian, NHST would conclude that native Hawaiians aren't US citizens."
So what is the observation in this case, and what would the corresponding prediction from the null hypothesis be? The observation that "0.2% of the US population is native Hawaiian" has no relation to your claimed null hypothesis at all.
The rest of your objection seems like one of those confused arguments trying to rule out basic reductio ad absurdam ("but if X really isn't true, then your arguments about seeing or not seeing the consequences of X have no basis!").
(And the central limit theorem has nothing to do with null hypothesis testing: you can do NHST with completely non-Gaussian statistics.)
You don't seem to understand my argument, so let me rephrase: The example about Native Hawaiians was meant to highlight the nature of conditional probabilities, and p-values are conditional probabilities. Just because p-values are below some threshold doesn't necessarily mean that the null hypothesis is incorrect and therefore should be rejected. Just because p-values values are high doesn't mean that the null hypothesis should fail to be rejected. P-values do not theoretically give that information. It doesn't even represent the probability of observing that value, since it's a conditional probability - as in, the probability of observing that value given that the null hypothesis is true, not the probability of observing that value. If the p-value is below 0.005, can you scientifically, theoretically conclude that the null hypothesis should be rejected? The probability of seeing a Native Hawaiian person given that the person is a US citizen is below the threshold of 0.005, but does that mean the conditioned part (the US citizen thing) should be rejected? Granted, it's hard to relate that example to actual experiments, but my argument is that p-values don't theoretically give any conclusions either way, and trying to make it "scientific" to draw conclusions by introducing thresholds to a conditional probability, no matter how strict, seems inherently flawed. Using it as a single metric among many, to use it as a tool for exploration makes sense to me. Even to make strong suggestions, sure, especially with all the controls RCTs put in. But to make hard conclusions, as in NHST? The approach itself doesn't have the theoretical power to do so.