What we need is a system designed to deal with this better. My suggestion would be greater incentives to the publication of negative results, and incentives to reproduce research. Create a system that encourages reproduction as a second step to peer review. This should also increase quality. If your research can't be easily reproduced it's probably not doing a whole lot of good even if ultimately you got the right answers.
Test driven research, if you will.
If everyone needs replication for publishing, the incentive to replicate other’s work increases.
Really? I don't know, "Don't be a fraud" might be a good Rule No. 1
I do think they shouldn't control that, but it's a deeper change, for a different gain.
Which goes to show the whole issue with this idea.
Journals do ask for exclusivity when they publish things. The don't ask for it just for submission. And the same reasons for why they don't now (nobody would submit anything to them) apply for the OP's idea.
In fact, all the changes only make the life of the researcher easier, because he would only have to present a proposal for the journals to deny. Now he has to complete his research before he gets the denial. (It also makes the life of the journals much harder, for the same reason.)
You contradict yourself. It absolutely changes things.
> Now he has to complete his research before he gets the denial.
It could degrade the incentive for the researcher to do great work, if they knew that they would be published.
Here an example of a typical project, where researchers are interested in the role of gene X in blood vessel formation in cancer, which could make it a good target for a cancer treatment. They want to knock the gene out and measure the blood vessel formation in an in vitro model.
Publication 1.
- We attempted to knock this gene out in this cell line, but due to an infection the cell line died. The installation was sterilized, the stored master cell bank was unfrozen, and we updated our sterilization protocols.
Publication 2.
- We attempted to knock this gene out in this cell line, but sequencing showed that there were off-target effects introduced. Reagents should be changed to be more specific.
Publication 3.
- The gene was succesfully knocked, but we found that a homologous gene appeared to activate the same pathway. The homologous gene should be knocked out as well to study the function of the gene.
Publication 4.
- The gene was successfully knocked out by disabling the gene and its homologue. The assay we used to study blood vessel formation was flawed. Due to insufficient oxygen supply, the positive and negative controls for blood vessel formation did not grow. New experiments should increase oxygen flow.
Publication 5.
- We successfully studied the function of the gene and found that it does not impact blood vessel formation in cell line x.
This is how the majority of research projects end up.
However, other researchers will be looking for a biological target to treat cancer X, and the information obtained by this series of experiments is not interesting. This is just one out of 99% of 30,000 genes that are not suitable treatment targets.
The conclusions are only interesting for researchers focused on gene x (and maybe its homologue), and potentially for researchers developing methodology. Questions answered were different than the original ones posed.
(I am paraphrasing Eliezer Yuskowsky here, specifically a small part of his NSFW and frankly weird collaborative fiction Mad Investor Chaos.)
Programming is such a wonderfully controlled environment, it's digital, replicable, and _almost_ math-like. You can know the variables. You can trivially replicate a program, or dig down into the OS... but every physical experiment out there has much more common and worse problems than Spectre or Rowhammer waiting to destroy it. It's the cosmic rays.
Your equipment/materials/specimens gets contaminated or reacts or ages. Your ultrapure water isn't. There's a vibration that changes everything. Your calibration sample changes. Cleaning staff used a new cleaner. The AC turned on. Build your experiment to be impervious to all of these and everything else I can't think of... or use the intelligence of an expert. Get there faster, cheaper, and find better ways of doing it along the way.
You'ld be right to say that it should all be documented, but the problem is that you basically need to be an expert to even understand the variables and to those experts many variables are so obvious to control they wouldn't think of doing otherwise. To give concrete and extremely valuable examples in manufacturing: EUV systems for lithography, OLED deposition and patterning, Lithium battery electrode forming. You can have the very best and exquisitely designed equipment for manufacture and yet without experienced experts running the process (replicating the experiment day in and day out) it will go out of control and you will get garbage in a million ways. It's like amateurs playing chess against a master... where the master is the natural variation of the environment.
High value semi manufacturing is a great example of how hard it is. The processes are designed to be as simple as possible to replicate. You replicate every day as many times as possible. You have huge amounts of money and resources to succeed and sometimes you still fail. Getting to 10% yield/replication is a big step... getting to 50% maybe harder... then 90%... and higher. Novel science is often harder and there's much less money and the time to replicate could be months/result rather than 1000s of testable results/min.
Perhaps with some justification, perhaps not. But if replication is impossible because of "irreducible complexity" (seems to be the argument?) then why bother with science at all?
Things are not that bad. But in some fields other approaches may be more productive.
For example clinical medicine may have more to contribute than scientific medicine. Psychological studies in particular are notorious for being impossible to replicate. Does not mean we should give up on psychology. It does not have to be science to be useful
Who's results would you publish?
1. Include the ones that passed and discard those that failed. 2. Run the tests several times and report only the best result. 3. Keep some of the details secret (data, some weird parameters, or maybe the whole repository), tweak something, run again, get some nice figures and publish a new paper.
Fortunately, he turned out to be right.
The reason there is "confusion" about whether many animal products are good or bad is because the corporations making billions of dollars off the product fund biased research.
Consider looking at the research digests from a non-profit (that sells no products, has no ads, etc) focusing on improving your health: https://nutritionfacts.org/
For instance this whole vegan trend is based on some radical religious opinion to control fertility and lower class reproduction by promoting a hormonal dysfonctionnal diet. All those from “official scientific paper” written by those religious fanatic few decades ago.
What a bold empirical claim. Could you share any evidence of this?
I reckon that lack of a good theory is the by far best predictor for non-replicable results.
Medical research in particular seems to suffer a lot from papers where something has been measured with no supporting theory whatsoever and consequently no real information other than "If you do what we did you probably get the same result". Rather unsurprisingly this is hard to replicate because there's no theory telling you what is and isn't relevant to replicate.
I'm not saying this is easy to fix, but maybe we should focus more effort on papers that expand the theory, rather than papers that at best show that something might be the case but we don't know why.
It does not make sense to act like small studies are proof of anything the way people like to use those studies on the internet. It is wrong that researchers are effectively punished for reproduction, verification etc. But the existence of small data collection studies on itself is not wrong.
One way to mitigate it is to pre-register studies as it's done in some areas like clinical studies.
Remove politics, free kidnapped and gagged science.
That needs the remotion of taxation. I agree.
Politics just adds some spin onto that. Education, I just don't think that's even a possible solution. The math involved in some of these fancier statistical techniques can be devilishly hard to get right, and it's often simply asking too much to expect someone to be an expert at both that and molecular biology. You could argue that that means people should look for competent co-authors to cover that side of things. But, see above - current incentive structures don't really favor the researcher who says, "I know! Let's bring on some nerd whose only job is to find reasons why we shouldn't publish!"
After all they are serving to the advance of science also (and saving thousands of expensive hours of research to other). If you want to promote it, just create incentives.
Otherwise, If the cheater wins in a 95% of the times and the verifier does not have anything to gain (and will damage their own research by the time invested on this) after a while only cheaters remain [1]. At middle term, all research tend towards a sort of expensive shamanism in well illuminated white rooms.
[1] (And of course the non cheaters will be actively blocked from this point).
Are they? If your proposed project does not align with current politics, no funding for you.
Nutrition? Would be surprising if there's much political distortion there but there's lots of really bad methodologies, partly because it's hard to get good data (people are bad at reliably writing down what they eat).
Social bot/misinformation research? Basically just politics masquerading as science. Methodologies are invariably bogus and conclusions are clearly determined before any 'research' is done.
Epidemiology? A mix of bad incentives and ideological distortion. The focus on policy advice isn't really scientific, and their policies somehow always end up requiring massive state intervention in people's private lives. This isn't something that falls naturally out of the science (which is unreliable anyway) but more the fact that they don't even consider any other possibilities.
The above three are pretty bad fields. There are plenty of better ones. Computer science is mostly OK. AI and cryptography are slightly dubious. Stuff like databases or computer graphics are fine. So IMO it doesn't make sense to talk about research or academia in general, or it does but only when discussing the universal incentive problems. Some fields just seem naturally resistant to political game playing because they don't have much/any impact on social policy to begin with. Obviously I mean politics as in democratic politics here, not the more mundane everyday departmental politics.
I can think of numerous examples of both from my career, from working with various colleagues and in stories I've heard. All of this is the tip of the iceberg I think: what you hear about usually are cases of blatant fraud, with high-profile or high-importance research. The bulk of greyer malpractice, with softer squishier levels of ethical dubiousness, and more indirect, cumulative impacts on peoples' careers and literature trajectories, are more hidden.
It's gotten to the point where I no longer put much stock in the reputation of individual researchers. Too many times I know in the best case there's an institution, or department, or subcommunity, or doctoral students or postdocs behind it, with who knows what sort of manipulation or luck. In the worst case there's years of fraud or other forms of corruption. Call me inappropriately cynical but I didn't start out this way.
I think we should also consider the possibility that some of it might be ideologically motivated.
For example someone who thinks people shouldn’t eat red meat due to climate change, animal rights, etc. (and who possibly belongs to a social circle where such beliefs are popular) might be motivated to fabricate results that could convince some people to eat less red meat (hypothetically, not saying that is the case here).