Reliable novelty: New should not trump true

If caught, the dishonesty would be truly obvious and harmful to a reputation

It would be difficult in practice to maintain the secrecy needed to pull this off.

While in many cases scientists currently keep what they're working on "mostly secret" until it's published, in fact lots of their friends and colleagues have a pretty good idea of what they're doing and would be in a position to ask hard questions if they tried to take your approach.

Doing that would require researchers to overtly and explicitly lie, which would act as at least some disincentive.

For short experiments you could game the system, but for an experiment that requires two years of work, it’s unlikely that you’d hold your results for two years to maintain the illusion.

Also you could tie the release of funds to publication of expected methods.

Wait, there are groups that don’t do this already?

“We thank reviewer 3 for the insightful suggestion...” (ok now copy paste the results that we cut for submission)

Ha ha only serious. Otherwise you get “helpful suggestions” like “run another few phase III trials first”

That would make it an explicit cheat. Right now, not publishing negative results, tweaking the procedure description a little and things like that lie somewhere in a gray zone and can be socially acceptable in some lab to some extent. However, if this would be explicitly defined, breaking this would be an explicit straightforward lie, which would make it harder to make socially acceptable in some limited circles, people would have to go to greater length to hide it and this kind of thing would not happen that often.

This is a risk, yes. I don’t necessarily think it’s a big problem.

First off, you’re describing premeditated, wilfully fraudulent behaviour. I suspect a lot of the problems are either less deliberate (you react to new information by making changes to your hypothesis or protocol without stopping to think what that does to your experiment’s reliability), crimes of opportunity (“this is so close, there’s no harm in fudging it a little bit, right?”) or a combination of both. It’s ok to enact rules that fight pervasive small issues while doing nothing about rare malicious behaviour!

Even the malicious behaviour would become much harder though. First off, the strategy you described only works for setups where the data collection stage is relatively short. If it takes a long time, then the fraudster’s timelines start looking suspicious. Second, that type of behaviour could really harm you if caught.

Hypotheses could be sealed upon filing. The existence of sealed hypotheses could/should be public, to prevent the system from being gamed by filing dozens of hypotheses at once for the same experiment and then selectively unsealing the one that matched their results. If the experimenter had too many sealed hypotheses to their name, that would be seen as cause for concern. But only a handful of sealed hypotheses would be unlikely to raise any eyebrows. Furthermore keeping one's sealed:unsealed ratio low would be an incentive to publish negative results. This system would also protect the experimenter from political pressure if they were testing a controversial hypothesis.

There's a long and unpleasant track record in science of people doing forming a hypothesis, doing trials, mining and changing methods and hypotheses and p-hacking for anything significant, and publishing a misleading paper that completely fails to mention that the original hypothesis was abandoned halfway through.

That may sound pretty agile. It is! You're absolutely right that this is an excellent match! The issue is that while agile may work pretty well for software dev, it perhaps doesn't work so well for science. It's given rise to a number of abuses.

It might also be noted that researchers doing experimental design are often quite familiar with their field, having gotten their hands dirty in a problem domain repeatedly.

There's the XKCD about significant P-values: https://www.xkcd.com/882/

To prevent that from happening, all the experiments that failed need to be reported too, and the link between them has to be obvious.

I don't know your field, but that hasn't really been my experience at all. Just about as much sloppiness all the way across the board.

My only systematic complaint here is that short format means much of the substance has to go in an inconvenient supplement, but that's hardly unique to Nature and Science.

They might be familiar with a subfield where there's a strong culture of accountability through reproduction of experimental results. Certain parts of molecular biology at least used to be that way.

Whenever I look at a biology paper in Nature I frequently see figures like this:

https://www.nature.com/articles/s41586-019-1112-8/figures/5

They’re utterly incomprehensible to my lowly engineers eye and seem deliberately designed to look as complex as possible.

What do these kinds of images and plots tell us of any value?