This would be a huge leap forward. I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. A null finding might not be great for the researcher but it's a win for science.
It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses.
While I agree that mechanisms like registered reports are the way to fix these things at the core, I do think that these mentalities ("it's just the system, I can't do anything but cheat with my scientific publications, my career!") are a big part of the problem.
Good read on the issue: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...
I get that. But the other side is that we pump up a lot of kids about science, then filter for very ambitious, very dedicated people and shove them into a resource-constrained environment where some kinds of results are much more strongly rewarded than others. It's absurd to set up a system like that and expect that science will be done to the level of frankness that you (and I) want.
I know some really smart, hardworking, very honest PhDs who have shifted to the tech industry because the roll of the research dice didn't come up right for them early in their careers. They are far from happy about that. I wish they could have stayed in science.
And I'd note that the person you're replying to is not "accepting scientific malpractice". They're saying they can't blame people for trying to survive in the system that they're caught in. If you want to be mad at somebody, be mad at the people who have set up the system, who could change it but don't.
It's unscientific and illogical to expect humans to sacrifice their lives on the altar of your belief: statistically, that's not what they're like. Especially in a system that filters out the most honest, least career-focused people early on.
This is a malpractice that effectively invalidates the research. But it doesn't feel so. It feels more like a thought-crime.
This is 'data mining' right? And I've occasionally wondered about this, since I don't work in a scientific field but did once make use of the scientific method for some research I did. And yes the findings weren't especially conclusive but I'm not sure I could've tweaked the hypothesis to make it work.
So, had I found something really interesting that didn't fit the hypothesis, is the 'right way' to conduct a new experiment from scratch? So say I did that, and used the 'tweaked' hypothesis, of course I'd find something interesting, because it's already there.
In this new 'pre-registration' framework, how can I correct the problem and pursue the interesting idea but keep the science in-tact? Because, if I used some sort of cross-validation at the outset and I have all the data available I presumably can't change the sample, so the hypothesis presumably has to change.
There are methods to account for follow-up experiments. Bonfaroni correction [1], for instance, requires you to increase your significance level with each new test.
On the other hand, other tests typically require the researcher to make explicit assumptions on the correlation structure of the experiments despite the fact that it is not directly observable.
Another more recent technique for 'exploratory' yet correct technique is to exploit differential privacy and dithering.
That would be datamining done wrong. Its perfectly fine to look at data to provoke new hypothesis. But you should not be using the same data to confirm the hypothesis that it provoked. Either use fresh data or make sure that you still ensure correctness if you are reusing the data.
If between tweaking the hypothesis and publishing it, you add the step "you perform another experiment which tests the tweaked hypothesis", you have just described the scientific method.
Though of course, there's a difference between "drug A doesn't work for condition B but seems to work slightly for C" and "drug A doesn't work for condition B but 10 of 12 individuals with condition C have shown significant improvement"
Most of the cases: yes, although your example of clinical trials is slightly different, and in that case, I do think the data should be publicly available to other researchers even in the case of a null.
In fact, you could replace [2] by a number arbitrarily close to 100% by increasing [1] accordingly.
You have an hypothesis, do an experiment, it fails. You mark the hypothesis false and move on, never putting the work into publishing it (why would you?).
At the same time, 19 other researchers have the same idea. Some 18 of them do the same as you, but one does the experiment and get a success. He will publish his work (why wouldn't he?), and it will be the only piece of literature available about the subject.
Where on this narrative did anybody do anything even remotely unethical?
Agreed in theory. In practice such don't even qualify - by definition - as science. Using the word science in such contexts gives such things far more credit and legitimacy than they deserve.
Likewise. The BBC reports that 2/3 of scientific research cannot be reproduced. Let’s be honest about what that means: 2/3 of scientists are fraudulent or incompetent.
No, it means that 2/3rds of the attempts to reproduce research fails.
When you work on the very edge of science, the lab you're working on it probably either the only one, of one of a very small number who are even able to perform the research that you're doing, due to a combination of highly specialised equipment, subject knowledge and just plain experience. Without that combination of factors, it's quite easy to mess up an experiment and therefore fail to reproduce the research.
I've never heard that. Is there some evidence that it's largely luck? Does the theory imply that Einstein and Newton were extremely lucky?
https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_inve...
For your everyday empirical research there is a lot of luck. You have to disprove a lot of possibilities to get to the true data. And as we only reward the positive findings your lucky if you’re test that one truth. If you’re sifting through drugs to find there possible other uses you’re going to have a lot of null results. But those are good they’re still extra knowledge.
Had this registration been in effect 300 years ago, Newton/Darwin would still be fine.
As to luck, who can say?
The fix for that would be to change science funding so a null finding doesn't hurt the researcher's career.
Let's say you try a particular educational intervention. it turns out that if has no effect on children's learning.
This is really useful information, because it means that we now know that there is no point in schools trying it.
There's something to be said for brute force on rare occasions though.
By publishing null results you avoid 100 scientists going the same endearing but unproductive routes, which can also help build good hypotheses and speed up progress.
A null result (no significant evidence for anything, a p value above 0.05) is truly null. It doesn't even confirm the null hypothesis.
Common wisdom says that Statistical Hypothesis Inference Testing doesn't work because researchers engage in "P Hacking". That's a half truth, researchers really don't understand the meaning of the p-value.
The stronger "absence of proof is proof of absence" would be fallacious.
A more accurate way to state this is “Absence of evidence is not always evidence of absence”. Or “Absence of evidence may or may not be evidence of absence depending on the circumstance”, though it doesn’t roll off the tongue as easily.
"When trying to reject the null hypothesis (sensu Fisher), absence of evidence (for its invalidity) is not evidence of absence (of its invalidity)."
Fisher wasn't the only statistician, he wasn't the only confused one either. We can go with Neyman and Pearson instead:
"When trying to reject the null hypothesis with a sufficiently powerful experiment, absence of evidence is evidence of absence."
This requires a power analysis, which is something most studies lack. Constantly reminding everyone of these details and misguided philosophical differences is tiresome, so I prefer to go with Laplace, Bayes, and Jaynes:
"Everything is evidence."
Today, most science is funded by government agencies, like the NIH. This means, bureaucrats who know nothing about science allocate other people's money to research they don't care about. Since they don't understand the research, they need objective measures of "research quality", and they picked number of publications and impact factor of the journals as measures. What you measure becomes your objective, and we got "publish or perish" and all sorts of scientific misconduct.
Science needs a free market, too.
Five million euro of tax payer money have been blown on a Neanderthal genome. All we got out of it is a hyper sensitive analysis that picked up an artifact (source: first hand experience with the raw data) and called it admixture. But it's a high profile publication thanks to the catchy headline and more tax payer money flows in the same direction.
This wouldn't have happened if the bureaucrats who presided over the grant money knew anything about the science and looked into the methods. Or maybe it would have happened anyway, because this was never science but just a publicity stunt. Either way, funding "science" this way is bad.
A great introduction to the topic and a great general guideline. It's not a collection of recipes, more like a nudge in a direction where you don't need canned recipes anymore.
"Surely that only works if your prior belief in your hypothesis is neither exactly 0 nor 1?"
And that's correct. Prior distributions are never completely concentrated at either extreme, for those could never be updated. If your hypothesis is now totally wrong, your confidence in it (the posterior probability) will converge toward zero.
In other words, its generally much easier for your head to come out of the barrel without an apple.
I get what you're saying, but the counter to that is it would be too easy for scientists to conjure up studies that they have no reason to believe is true, do an experiment, publish the null finding (that frankly, they were expecting), and ask for more money.
It's almost trivial to think up experiments that will give you a null finding.
Unless of course you want to reward null findings which might, depending on the circumstances, open another can of worms.
What about a lowly restaurant-owner who skimps on food quality or sanitary costs to "help their career?" A psychopath or "can't blame them?"
Erroneous research can also have human costs.
White collar and blue collar workers should both be scrutinized equally.
I'm not sure where this acceptance of moral depravity among more-privileged peoples comes from, but I really don't like it.
For example, a big part of how the file drawer effect happens is that writing papers is expensive and time consuming, and it's hard to get anyone to publish negative papers, and academics' careers operate under a rather brutal "publish or perish" regime. All that adds up to, if you get a negative result, you've got a whole lot of very concrete reasons to cut your losses and move on. The goody two shoes who's scrupulous about reporting all their findings is not going to get rewarded for their efforts with a job. Nor will they be rewarded with the satisfaction of knowing they've done their part to improve the average quality of the published literature. Out-of-work scientists don't get many opportunities to do research.
This is the most-downvoted comment I've ever made on HN, and I think your explanation is in summary what people disagree with.
However, the problems of bias and integrity in scientific research can and do have costs in terms of harm to human life. It's just that the connection between just following incentives and bad scientific research is much more abstracted, and therefore is not clearly intentional negligence, as the case with something like food safety.