You got a null result. Will anyone publish it?
nature.com
nature.com
Our editor-in-chief was interviewed for this related Nature article a couple months ago (https://www.nature.com/articles/d41586-024-01389-7).
While it’s easy pickings, it’s still always worth pointing out the hypocrisy of Nature publishing pieces like this, given that they are key drivers of this phenomenon by rarely publishing null results in their mainline journals. They are have extremely little incentive to change anything about the way scientific publishing works, as they are currently profiting the most from the existing structures, so them publishing something like this always leaves a bit of a sour taste.
Because of this revealed lack of interest, high tier journals don't tend to take of them (correctly responding to the lack of demand), and journals like your that specifically target these kinds of articles A) struggle to succeed and B) remain relatively "low impact", which means that the professional rewards to publishing in them are not very high, which means that the return on effort of publishing such a work is lower.
Don't get me wrong, the scientific community could do a lot more to combat this issue, but the core problem is that right now, the "market" is just following the incentives, and the incentives show that, despite what the stream of articles like this one over the past few decades is that most scientists don't seem to actually have an interest in reading null-result papers.
I would love something like Living Papers [0][1] to take off, where the null an non-non results could be compared interactively on similar footing.
A null result should be published right away in a searchable place, but probably isn't worth a lot of effort in general. I tried X, it didn't work, here is the raw data.
Even if the whole thing is a null, the setup, instruments, dependencies and what methods worked/didn't work is worth describing by itself.
Papers need to be published backwards.
* the current process still has a lot of luck in terms of getting assigned referees. Sometimes you just plain get a bad reviewer who just can't be bothered to read the submission carefully and is quick to reject it. I would hate to see a system that only allows for a single shot at publication
They should even publish it before the results arrive.
And then publish again after.
If you think it will be helpful to others to know about specific failure, put it in a blogpost or even on arxiv. Or talk about it at conference (for CS, workshop).
Also, if we use publications as a measure of scientists success, and we do, is a scientist with a lot of null results really successful?
That is a non-ideal way for the scientific community at large to maintain such info. Trying to go through traditional peer review process is probably also non-ideal for this type of work though, for reasons you cited. We need to be willing to look at publication as something more broadly defined in order to incentivize the creation of and contribution to that sort of knowledge base. It shouldn't be implemented as a normal journal just meant for null results - there's really no need for this sort of thing to be peer reviewed specifically at the prepub stage. But it should still count as a meaningful type of scientific contribution.
And in the very old days, newspapers used to publish in tabular form local election results and sports games.
I feel that Nature could dedicate one to two pages of one paragraph summaries of null results with links to the published papers.
It's amazingly easy to skim such pages to find interesting interesting things!
[1] I think that was the name; I canceled my subscription when they changed to a Scientific American wannabe. I was looking for breadth not depth! I could always get the original paper if I wanted more information.
[1] says:
> In statistical hypothesis testing,[1][2] a result has statistical significance when a result at least as "extreme" would be very infrequent if the null hypothesis were true
So I understand this journal publishes results for which a hypothesis was tested, found to give insignificant results, which would rule out the hypothesis assuming the research was correctly conducted, without biases in the methodology, with a big enough sample, etc. Which would be worthy to know but no journal usually takes this research because it doesn't make the headlines (which yes, I've always found was a shame).
Do I get this right?
>Which would be worthy to know but no journal usually takes this research because it doesn't make the headlines.
That's usually correct, which gives rise to all kinds of issues like the article talks about. It can result in a lot of wasted time (when you're conducting what you "think" is a new experiment, but it's been done many times but unpublished because it doesn't provide statistically significant results). It provides little incentive for replication, which can lead to stronger conclusions about the results than may be warranted, etc.
There’s so much to learn from these cases.
One of our goal is to change the perception of and culture around failure in academia. Research/science is not just a steady upward trend of progress, it comes with a lot of trial and error. Academics’s success and job prospects however depend mostly on them publishing in high impact journals, which in turn only publish “interesting” aka positive results, which creates this very toxic publish-or-perish culture. Having an experiment fail is a natural part of doing science, but academic institutions punish you for not producing positive results. By providing a place to publish these failed experiments, at least it provides some relief for this problem. This is not real change however, that needs to happen at a much higher level, but that is one we do are not able to impact. Ideally our journal will not be necessary in the future, as we detail in our opening editorial: https://doi.org/10.36850/ed1
Choosing to only publish the most extreme positive values means that when the experiment is replicated, "regression to the mean" makes it very likely that the measured effect will be weaker, and possibly not statistically significant. This is not an evidence of scientific fraud -- rather, it is a predictable outcome of a publishing incentive scheme that rewards hype and novelty over robust science.
I've said it before but it bears repeating - replicating published results, and adding the findings to a database, should be a standard part of PhD training programs.
I’m not saying that our current allocation of resources is optimal. I am pointing out that our resources are finite and “replicate everything” is not even a remotely practical allocation of those resources.
GP is pointing out that the incentive structure makes this an invalid assumption. If publications reward hype and novelty when deciding what to publish, then there is no point spending your limited resources replicating other peoples' results, they won't get published anyway. And experiments that give a null result won't be published anyway. What's left are one-off results that showed something surprising simply by chance and don't replicate...but then, we generally will never know that they don't replicate, because the replication experiment is not novel, has a low chance of being published, and hence isn't worth spending limited resources on.
Basically the publication process introduces selection bias into the types of research that are even attempted, which then filters down into the conclusions we take from it. A cornerstone of the scientific method is random sampling, but as long as the results that get disseminated are chosen by a non-random process, it introduces bias.
I don't know that this is accurate. Some of these make their way to large-scale public policy, or give bona fides to people who craft far-reaching policy. This includes changes to 401k allocations to car-insurance rates and other mundane, but consequential policies.
The truth is most of science is not important outside of popular science news. So we shouldn't be surprised that the bulk of replication crises are also in the same category. Claiming this means the replication crises is not really impactful may be a case of base rate neglect.
It's also important to note that your example of medical replication is a relatively highly regulated area, where most other science is much less so.
The example you provide is solely your assumption? Seems pretty odd to provide a baseless assumption as an "example".
Undergraduate students love being involved in research. It's one of the selling points of many top universities. Grad students replicate research all the time. Maybe funding is an issue in grant fields (some research is extraordinarily expensive) but that doesn't excuse the lack of replication across the board.
Yet we still pool scientific results using only the printing press.
It's like we unlocked the tech tree but then got so caught up in chasing citations and peer review that we forgot to use the new tech we invented.
Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. Only publish positive results, and someone will eventually publish a false positive result without fraud.
You need different samples. There isn't a statistical trick that gets around this.
For example: let's say there's a cancer with 20% survival rate. You test a treatment with 25 experimental and 25 control patients, 40% in the experimental group survive[1].
You can analyze this with a bunch of statistical methods. You can ask different questions about the patients, focusing on well-being rather than simple carcinogenic remission. But ultimately, the thing that happened in this study is that 50% got better and no fiddling with numbers or changing the questions you ask is going to change that underlying phenomenon. You can check for blood markers of cancer: you get 50% have no blood markers. You ask them questions about how they feel: you get 50% feel better. You body scan the area where tumors were: you get 50% no longer have tumors.
You have only tested one phenomenon in one sample, and that essentially amounts to 5 people getting better.
[1] I know this is not how cancer treatment studies work exactly, this is a simplified hypothetical.
Eventually if we (the public) are lucky someone else in the field will disagree and run the trial again, which is how you get the alt text.
Particle physics still uses five sigma as the significance threshold.
I think most people correctly intuit that this is actually a type of very pernicious fraud.
This seems like such an obvious problem in the way science is currently done. Are people so focused on their own individual fields that they aren't thinking about and fixing such glaring meta problems?
Consider: "Hey, look, I went on top of tower of Pisa and threw down two identically shaped balls, one iron and one wooden. They dropped at the same time!"
The above is the expected result and would only be interesting if the result is different from expectation. Now, if 1000 scientists did this and each published the confirmation of what we knew would happen then who would read that? But, if one scientist said: "I tried and they drop at different times!" that would be different. The 999 scientists would then try to replicate again and then the papers of the 999 would be interesting again.
That would be true if p were the probability of null hypothesis being true given the data observed, but that’s not what it is.
https://eurradiolexp.springeropen.com/articles/10.1186/s4174....
This is the sort of thing I used to do when I was younger, and looking back, the reason I did it was because I was basing my sense of self-worth in being smarter than other people. Ironically, this made me dumber, because I was less open to the possibility of being wrong and therefore was slower to learn. And, I found doing this made people dislike me.
I think you're interpreting that sentence as talking about one false positive out of 20 positives? Which would be a very incorrect statement.
But it says one false positive out of 20 experiments. That's a valid statement. (Though it does depend on the proportion of negatives you're getting.)
And knowing all this, this behavy should be considered borderline scientific fraud at this point.
Wait, why should PhD students do that work? That just sounds like pushing more grunt work to the lower rung of the academic hierarchy.
Nope. If you want people to do that kind of work that is important to everyone but is not directly conducive to promoting one's research career then the solution is simple: pay them.
I think there is some reasonable argument that replicating research is the first step to learning how to do good research on your own. In an ideal world, PhD students should probably be trying to replicate similar work anyway and applying existing approaches own pet problem. In practice, many gloss over this because they are narrowly focused on doing something "new" so it can get published.
I just got my PhD last July and that sounds like boring drudgery rather than "good exercise". Good exercise is to have a student write their own paper, dig up their own references, formulate their own experimental hypotheses, run their own experiments, write up their own results etc. Re-doing someone else's possibly badly-done work is grunt work that should not be forced upon anyone.
>> PhD students ARE paid.
Haha. Good one XD
In its scriptures/philosophy, science describes extremely thorough and sound principles and guidelines...but in on the ground practice (by scientists, which are a part of "science"), they are often not achieved[1]. However, this distinction is not only not advertised broadly and without aversion, it is usually (in my experience) not mentioned at all, if not outright denied using persuasive rhetorical language (like, for example, when an object level instance of not achieving it is pointed to in the wild, such as in forum conversations). This may not be fraud (that requires intent I think?), but it achieves the same end: misinforming people.
I absolutely agree with your database idea, and if science would like me to take them seriously (something near how seriously they take themselves) they'd also have to go much further.
[1] Not unlike in religion, a competing metaphysical framework (model of reality) to science.
In your footnote, for example, you translated your philosophy-speak into English (metaphysical framework -> model of reality). Why not just say that? Your entire comment goes into "philosophy mode" and communicates a few very simple ideas in overcomplicated language.
Science and religion are pretty poorly understood as competing models of reality. Religion originates when people make up answers to other people's questions to gain social standing, and religion continues due to (among other things) anchoring bias--the bias people have toward continuing to believe what they already believe. While religion does result in those people having a model of reality, there is no attempt being made at any point to relate the model to reality. When religious people and scientists disagree, it's not because the religious person is trying to model reality differently--the religious person isn't even trying to model reality--it's because the religious person is biased in favor of their existing belief.
You said:
> In its scriptures/philosophy, science describes extremely thorough and sound principles and guidelines...but in on the ground practice (by scientists, which are a part of "science"), they are often not achieved[1].
This is presented as some sort of gotcha, but it's not: few scientists will claim that science is being practiced perfectly or even well. Outside of a few areas such as particle physics, we're quite aware that our ability to practice scientific ideals is hampered by funding, publication incentives, availability of test subjects in human studies, data privacy, etc. And we're aware that this means that our conclusions need to be understood as probabilities rather than 100%-confidence facts.
There are certainly some people who treat scientific conclusions with religious absolute confidence, but doing that is fundamentally against scientific principles. The accusation you are leveling against science would be better targeted toward people: generally science journalists and the science-illiterate public rather than scientists themselves. The entire reproducibility crisis is scientists using science to show that our practice of science is too imperfect to result in high-confidence conclusions.
Religious people jumping on the replication crisis because they think it disproves science is rich. The replication crisis isn't a disproof of science, it's an application of science. The reason we know that there's a replication crisis is because scientists asked "How confident can we be in the conclusions of existing studies?" and applied science to answer that question. If you really think science is invalid, then you can't use science to prove that.
And the fact remains that any confidence in conclusions at all is more than religion has to offer, because again, religion isn't trying to model reality--the fact that religion produces a model of reality is merely an unfortunate side-effect.
A problem: you're talking to one right now, and you (your mind's model of reality, technically - you do not have access to the state of the things you claim to) could hardly be more wrong.
From large quantities of experience, I am confident I would have no success tackling your disagreements on a careful, strict, point by point basis. Instead, I will simply present two links (I have many others, but let's see what happens with these) and ask: do you believe these have some substantial relevance here, related to the truth value (and appearance of) of our respective claims?
https://en.wikipedia.org/wiki/Theory_of_mind
https://en.wikipedia.org/wiki/Direct_and_indirect_realism
I can appreciate that this approach may seem unworthy of anything more than a rhetorical response that dodges the question (and the importance of the phenomena these links discuss), so hopefully you can take the challenge seriously. I am more than happy to offer a more substantive reply later, but if you declare victory by fiat[1] it's a bit tough to have a serious conversation.
[1] Roughly: declaring that one's opinion of the unknowable is necessarily correct, and that it is(!) contrary to my (actual) stance.
Anything you're going to say to try to show that religion is trying to model the world is going to be an attempt to use logic and observation, because those are the tools that work, and you know those are the tools that work. You're just using them with an end goal in mind due to aforementioned anchoring bias, which isn't really the way to use them.
What you probably aren't going to do is tell me to pray and God will tell give me the answers, or tell me to read a book written by ancient people who believed stuff like "the sky is a dome", because you're well aware nobody takes that seriously as a way of modeling reality. I mean, maybe you are going to tell me to do those things: if so, that's embarrassing.
And the latter is what religion has to offer for modeling reality that science doesn't have. Sure, religion uses logic and observation because they so obviously work, but science has logic and observation too, and frankly, is better at it--we don't need religion (or philosophy) to tell us how to logic or observe.
So if you really want to present religion as a way to model reality, that's what you've got to prove has value for modeling reality: talking to invisible friends and consulting outdated writings. We're already convinced that logic and observation work: how do your weird additions compete with that?
A problem: you're talking to someone who used to be religious, so I have as much access to the internal thinking of a religious person as you do.
A second problem: people's self-perceptions of their own internal processes are quite often measurably wrong.
> From large quantities of experience, I am confident I would have no success tackling your disagreements on a careful, strict, point by point basis. Instead, I will simply present two links (I have many others, but let's see what happens with these) and ask: do you believe these have some substantial relevance here, related to the truth value (and appearance of) of our respective claims?
> https://en.wikipedia.org/wiki/Theory_of_mind
> https://en.wikipedia.org/wiki/Direct_and_indirect_realism
Short answer: not in any interesting way.
Long answer:
From large quantities of experience, I would guess that you're about to make a special pleading argument that based on convenience beliefs that you yourself don't believe in any other context, as evidenced by the fact that you don't practice them.
Those pages, particularly the latter, are another example of poor communication being presented as intelligence. If we translate to English instead of philosophy-speak, it boils down to an argument about whether perception is reality or not.
Let's cut to the chase with a relevant parable:
The Buddha and his disciple were walking down the road. Suddenly, the disciple drew his sword and cut the Buddha in half at the waist. The Buddha turned to his disciple and said, "Now you're beginning to understand!"
Would you be willing to reproduce this parable experimentally with you as the Buddha? After all, perception is reality, so if you're the Buddha and you perceive being cut in half with a sword as no big deal, that will be just fine, right?
The thing is, science is perfectly capable of answering this question--it's not unknowable. The experiment of cutting someone in half with a sword has sadly already been performed too many times in history: we don't need to perform it again. The scientific answer, which we already have, is that no amount of changing our perception prevents the person cut in half with a sword from dying in horrible agony. And when you're not speaking philosophese, you already believe the scientific answer just like every philosopher who believes perception is reality until faced with the prospect of being cut in half with a sword. So if you're about to make an argument about direct and indirect realism, I'd have to ask, why do you believe that reality is reality when it comes to swords (and everything else in your day-to-day life), but you suddenly you want me to believe that perception is reality when it comes to your invisible friend?
My only opinion of the unknowable relevant to this conversation is that by definition, neither of us knows it.
More parts of philosophy I think we can discard without losing anything of worth:
1. Arguing that perception=reality when it's convenient and refusing to practice it in any other context.
2. Talking about the unknowable as if we know it.
Just one way you've missed the mark: you are speaking as if your access to my mind is as good as it is to yours.
> Short answer: not in any interesting way.
To you, or to all people?
> From large quantities of experience, I would guess
At least here you realize you're guessing.
> that you yourself don't believe in any other context, as evidenced by the fact that you don't practice them.
How could you possibly know this?
> Those pages, particularly the latter, are another example of poor communication being presented as intelligence. If we translate to English instead of philosophy-speak, it boils down to an argument about whether perception is reality or not.
Here you are speaking as if you perceive reality as it is directly, no interpretation, no bias.
> Let's cut to the chase with a relevant parable:
> After all, perception is reality, so if you're the Buddha and you perceive being cut in half with a sword as no big deal...
It may be relevant, but it certainly isn't representative.
> ...that will be just fine, right?
Incorrect. Might you be thinking in binary?
> The thing is, science is perfectly capable of answering this question--it's not unknowable.
"Science" only has volition through scientists (or "scientific thinkers"), and Humans very often are not able to not know things (particularly educated Humans at times....a little knowledge is a dangerous thing as they say). Studying science does not turn one into a Perfectly Rational Human, after all.
> The experiment of cutting someone in half with a sword...
Look at you milking this incorrect strawman!
> you already believe
You can not read my mind. You are literally hallucinating.
> So if you're about to make an argument about direct and indirect realism, I'd have to ask, why do you believe that reality is reality when it comes to swords (and everything else in your day-to-day life), but you suddenly you want me to believe that perception is reality when it comes to your invisible friend?
Please, tell me who "my" "invisible friend" is, Human. Tell me in vivid detail, I would really like to know this about my actual, not indirect self.
> My only opinion of the unknowable relevant to this conversation is that by definition, neither of us knows it.
Note that this itself is an opinion (an opinion of an opinion).
It is also a tautology, which in this case I would classify as cheating (you are making claims to know the unknown, and backing it up with a tautology).
> More parts of philosophy I think we can discard without losing anything of worth
When you say "think", I'm curious what you mean. Could you (after the fact, giving you an advantage) write out a pseudocode representation of the chain(s) of logic (and the axioms & premises upon which it is based) you ran to arrive at that incorrect conclusion? Because I have a sneaking suspicion there might be some heuristics and bad axioms(!) in the cognitive pipeline.
> More parts of philosophy I think we can discard without losing anything of worth:
> 2. Talking about the unknowable as if we know it.
Ok, this is too much. Have you been putting me on this whole time?
I'll shoot the fish in a barrel in your followup later on (I am "posting too fast"...what on this site isn't at best a half truth?).
2. I said I was guessing, and it's possible my guess war wrong. But frankly, nothing you've said actually indicates that my guess of your point was wrong--it simply indicates that you didn't understand my post. So I'll ask you point blank: do you believe perception is reality? A simple, "yes" or a "no" will suffice, no need to flower it up with philosophese.
3. You say you're a religious person. Which religion? I doubt you believe the vague, slippery, undefined religion that exists only to allow religious people to debate atheists without having to defend the absurd beliefs of their actual religion.
> Just one way you've missed the mark: you are speaking as if your access to my mind is as good as it is to yours.
No, I'm saying my access to the mind of a religious person is as good as yours.
> > The thing is, science is perfectly capable of answering this question--it's not unknowable.
> "Science" only has volition through scientists (or "scientific thinkers"), and Humans very often are not able to not know things (particularly educated Humans at times....a little knowledge is a dangerous thing as they say). Studying science does not turn one into a Perfectly Rational Human, after all.
Yep. Which is why I didn't say "science is perfectly capable of answering ALL QUESTIONS". What I said was "Science is perfectly capable of answering THIS QUESTION." Since you're intent on taking everything out of context, I will insist on clarifying that what I mean is, "Science is perfectly capable of answering the question of whether perception is reality or not--whether perception is reality or not is not unknowable."
I will go on to say, religion isn't capable of answering any questions. No criticism of science actually gives any validity to religion.
> Please, tell me who "my" "invisible friend" is, Human. Tell me in vivid detail, I would really like to know this about my actual, not indirect self.
Your invisible friend is whatever deity your religion believes in. You said you were religious.
> When you say "think", I'm curious what you mean. Could you (after the fact, giving you an advantage) write out a pseudocode representation of the chain(s) of logic (and the axioms & premises upon which it is based) you ran to arrive at that incorrect conclusion? Because I have a sneaking suspicion there might be some heuristics and bad axioms(!) in the cognitive pipeline.
Bro, what are you even asking? If you want to know why I think philosophy is often poor communication presented as intelligence, it's crap like the above quoted paragraph.
No, I'm not going to pseudo-code out the definition of the word "think", if that's what you're asking me to do.
You say you "think" you've made your point. Remember when you said:
> When you say "think", I'm curious what you mean.
Strange that you suddenly know what that word means.
No. Correlation fallacy.
Naive Realism fallacy.
All results should be published, some should be celebrated.
Some people publish fantastic papers without data nor code. Sometimes annoying, other times a complete waste of everyone's time.
In a world where they are decoupled, you'd only end up reading it because somebody you trust endorsed it (no reason for that to coincide with the publisher) or because you went searching for it despite its lack of endorsement--in which case you probably know what you're doing.
I was specifically talking about peer-reviewed papers and how bad some of them are. I'm working in the space of parallelization and optimization and a lot of papers out there just implement any given algorithm on any hardware of their choice. They describe vaguely what they have done, claim to be faster than any other papers and call it a day.
That's worthless in my opinion. For all we know they could have a bug in their code and that's why it's fast. Some may even lie, who knows without code or data.
But oh well, I guess it's a step towards graduation for them. So at least they got something for their time (and presumably they have that cool, fast implementation they need for a project and some useful experience, but the paper: worthless).
While the resemblance of my A chart with their B chart might be a bit subjective, I have a reproducible build of my chart and its labeling that I can trace all the way back to the source data (via nix). It might not be perfect, but at least there's an artifact available for scrutiny... at all. The published version though, it's just a PDF. I can't see their code so that I can point them at the bug that I suspect exists in it. Their data is available on NCBI, but not their code.
So yeah, I hear you re: being a bit disappointed with the reproducibility of the computational aspects of these papers. It's somewhat understandable for replication to be difficult in places like psychology or chemistry... but if the work is being done in silico, it should be a matter of two or three commands to re-do it in silico. And if that's not true, we need to fix the tooling until it is.
I'd prefer to have the raw data, intermediate artifacts, and their computational relationships, all made available as soon as the researcher has an inkling that they might be worth presenting to someone. Then, when it's time to celebrate those results, they can just reference things that have been publicly searchable and verifiable for a while now. That way computational verification can happen before the paper is even written, and we don't even have to bother writing prose around what turned out to be bogus results.
I realize as I'm writing this that I don't really understand what "publishing" means. It's more than just making the paper available, right? Is there a formal definition, or just a colloquial one in science?
When this works well, it’s a good filter to prevent spam, fraud, methodological errors, etc. from being published, while improving the quality of the accepted research papers via feedback from other domain experts.
When it doesn’t work well, the referees can take it upon themselves to reject papers for subjective reasons, including that the work is “not novel enough”, that they don’t like the model you used, or that they are just not excited by the research field you work in. It also happens that they require you extend your work in a way that takes an order of magnitude more time before they’ll accept it. For the authors, it’s often difficult to defend themselves from this kind of attacks, since the referees in many journals don’t need to justify their claims much, and often feel free to be extra harsh since they tend to be anonymous.
Since going through the publication process can take months to years of work depending on your field, some researchers would not be willing to put in that effort for a negative result (which is unlikely to be cited and thus doesn’t help your career).
It is however possible to just upload a paper (e.g. to arXiv). These “manuscripts” are often useful and can be cited normally, but researchers tend to be a bit more wary of citing them unless the authors are well-respected due to the lack of peer review.
1. Fully gather and analyse the data, no stopping early when you realise it isn't working
2. Write the paper, read those background papers you hadn't got to yet so you can cite them, chase down references for things you know from memory.
3. Realise there's a gap in your table because you tested A, B and D at three levels each but C you only tested at the low and high level, not at the medium level. Go set up your test equipment again to fill in the blank space.
4. Run the paper by your collaborators and your boss, all of whom will feel obliged to suggest at least some improvements, which you'll make.
5. Choose a journal, apply the journal's template and style, send it in.
6. Wait for as much as several months for peer review.
7. The first peer reviewer suggests you retest with a slightly different protocol for cleaning your equipment before the test. You do so.
8. The second peer reviewer replies suggesting you test combinations of A, B, C and D, not just one at a time....
What are the corresponding statistics for researchers that find positive results? Closer to 100% are willing to publish? And how many succeed?
Not everyone is so fortunate. This lesson has stuck with me, as I have seen or heard from different labs where, unfortunately, they couldn't afford to publish their findings.
This happens because hypothesis testing conflates effect size (how big is the difference between A and B) with uncertainty about that effect size (significance/reproducibility). Confidence intervals are more useful IMHO, as they help untangle these two aspects, for example showing that the difference between A and B is small _and_ reproducible. Bayesian analysis is also a major improvement, as it allows examining both the "null" and "alternative" hypotheses on equal terms, as well as reasoning about our prior beliefs / biases. Unfortunately many areas of science are still stuck with statistical methods from the early 1900's.
Imagine the corresponding concept for programmers: you are not allowed to sell or share any software you create unless you pre-register a detailed plan for what code you will write and how it will be used before you write the first line of code. Pretty sure that would reduce the innovation going on in public GitHub repos a lot :)
Maybe pre-registering isn't the right answer; I'm sure there are practical hurdles, but the problem to be solved still remains the same (visibility into the graveyard of failed experiments to improve the rate of innovation).
It's not like it couldn't be gamed, but maybe it would incentivize people to also publish null results.
This will hardly achieve this goal as you basically make things harder, and now you are introducing more overhead. The main reason why people don't like publishing null results is that it hurts them in funding applications. The current system works with mentality, we shouldn't fund someone who don't get positive results. It is better to allocate this somewhere else. Most of the problems with research can be tracked down to funding issues and practices. But these are political issues, so people try to argue about other things because it is easy.
https://en.wikipedia.org/wiki/Metascience
And read "Why Most Published Research Findings Are False".
https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...
Also, just because you get a null result doesn't mean that nothing was learned, that something new (and unexpected) wasn't stumbled on, or that some innovation didn't happen.
There are tiers of publications and journals. Even if you get a null result and you're not going to get it accepted in Nature, it's very possible that you can get a conference paper (sometimes peer reviewed) out of something that was learned.
How do you distinguish a ‘real’ null result from one done in a sloppy study?
Would people run shoddy experiments to get null results to undermine their rivals?
Could somebody pump out dozens of null publications to pad their CV and screw up h-indexes?
1. Ideally peer review would catch this. A badly setup study should be critiqued in peer review. Forcing scientists to first publish their methods before doing the experiment also helps, as it validates the experimental setup before hand.
I also think it’s worth publishing studies where a null result was reached due to some error in experimental setup or other factors, as long as it’s presented as such and reflected upon. This can still be valuable information for future experiments. Offering scientists social capital for that (an “official” publication, citations) might also incentivize scientists to publish the results as is, rather than making it appear as a “true” null result, or even as a non-null one (eg through p hacking).
2. While obviously possible, given the amount of effort scientists have to go through to raise funding for an experiment nowadays, i find it highly unlikely that people would go through this effort.
3. This is already possible and a problem. This is a problem of academic misconduct, has very little to do with null results.
The current publishing system is of course already set up to be gamed, so I understand your worries. But null results should be published, as they are just science. Even if someone were to “game” the system by publishing a ton of null results, those publications should be held to the same level of scrutiny as any other publication. If someone is extremely prolific in replicating existing studies and comes up with a ton of null results, that should be lauded and those papers should be published, no?
I do believe the entire idea of a researchers output only being recognized by being allowed to be published in a journal is terrible and should be abolished, but baby steps I guess.
>Would people run shoddy experiments to get null results to undermine their rivals?
In this case, the rival would be very much inclined to recreate the "null" experiment.
>Could somebody pump out dozens of null publications to pad their CV and screw up h-indexes?
Possibly, but would null publications be cited as often? Also, who's going to keep funding a researcher that mostly publishes null results?[0]
[0] Besides agenda-driven "think tanks". Which is worrying itself.
Don’t underestimate an academics ability to cite ALL of their previous publications each time they publish
This article completely glosses over the fact that to publish a typical negative result, you need to have progressed your scientific career to the point where you are able to do so. To get there, you need piles of publications, and since publishing positive results is vastly easier than publishing negative ones, everyone is incentivized to not waste time on the negative ones. You either publish or you perish, after all.
Simply put, within the current framework of how people actually become scientists and do research, there is no way to solve the 'file drawer' problem. You might see an occasional graduate student find something unusual enough to publish, or an already-tenured professor with enough freedom to spend the time submitting their manuscript to 20 different journals, but the vast majority of scientists are going to drop any research avenue that doesn't immediately yield positive results.
Positive result claim: the sun comes up in the morning.
Negative result claims: the sun moves sideways in the morning. The sun was always there. The sun peeks up in the morning and immediately goes back down. And so on.
Positive result claim: aspirin is an effective pain reliever.
Negative result claims: eating sawdust is an effective pain reliever. Snorting water is an effective pain reliever. Crystal Healing is an effective pain reliever. Etc.
Because there are so many negative results, it's trivial to construct an experiment which produces one. So why should that be published?
Negative results should be published when people in the community are asking that question, or have a wrong belief in the answer (hence the replication crisis). But if nobody cares about the question, it's hard to argue for why a given negative result would be preferred over any other negative result for purposes of publication.
In the language of hypothesis testing you have your null and alternative hypotheses.
So for alternative hypothesis that the sun comes up in the morning, the null hypothesis would simply be that the sun does not come up in the morning.
Each of the negative results, reads to me like a separate 'alternative' hypothesis.
So let's say I claim that the sun goes in a circle in the sky in the morning. The null hypothesis is that it doesn't do that. Perform experiment. Null hypothesis wins. Write up paper! This is a negative result.
The point is that for every result where the alternative hypothesis wins, there are a massive, if not infinite, number of results where the null hypothesis will win. Are these publishable?
> It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so.
??? As I had said originally, that's one of the primary situations where a negative result should be published.
But the huge, huge, huge majority of negative results are trivial and uninteresting. Thus the fundamental issue with negative results is that you have to provide rather more compelling justification for why such results should be published.
In my opinion the goal is to get a record of the information and the dataset out there.
Yeah, that's not science - it's the exact opposite of science. This is the perfect example of why reasoned skepticism is more necessary than ever. Blind trust in any institution is a recipe for disaster.