Hacker News Reader

Comment by jedberg | Hacker News Reader

I'm surprise the solutions section didn't include the most obvious solution of all: require the hypothesis, procedures, and expected results to be published before beginning the experiment. This would force them to publish their results if they fail, and also publish if they changed their methods and why. It still wouldn't be 100% reliable but it would help.

The other solutions presented were also good though. Making the data cleaner for other to analyze is always a good thing, especially in the "machine learning" age.

View on HN

This has been discussed before:

https://dynamicecology.wordpress.com/2013/10/21/two-stage-pe...

http://neurochambers.blogspot.com/2013/04/scientific-publish...

Personally as a researcher, I think the idea is definitely a good one.

I believe this practice should be inculcated in most research projects. However, if I put my adversarial hat on - I see one challenge. Scientists could still perform their experiments beforehand with the hypothesis, procedures and results. During submissions, they will _phase_ these two-stage information and give an impression of not performing posthoc rationalization. However, in reality they would. Such a system could be business as usual for the scientists unless you control the power for scientists to perform their experiments which seems impractical.

The idea is extremely great but I believe this could be difficult to implement at a publication level.

If caught, the dishonesty would be truly obvious and harmful to a reputation

It would be difficult in practice to maintain the secrecy needed to pull this off.

While in many cases scientists currently keep what they're working on "mostly secret" until it's published, in fact lots of their friends and colleagues have a pretty good idea of what they're doing and would be in a position to ask hard questions if they tried to take your approach.

Doing that would require researchers to overtly and explicitly lie, which would act as at least some disincentive.

For short experiments you could game the system, but for an experiment that requires two years of work, it’s unlikely that you’d hold your results for two years to maintain the illusion.

Also you could tie the release of funds to publication of expected methods.

Global. Warming. Exponential. Curve.

Wait, there are groups that don’t do this already?

“We thank reviewer 3 for the insightful suggestion...” (ok now copy paste the results that we cut for submission)

Ha ha only serious. Otherwise you get “helpful suggestions” like “run another few phase III trials first”

That would make it an explicit cheat. Right now, not publishing negative results, tweaking the procedure description a little and things like that lie somewhere in a gray zone and can be socially acceptable in some lab to some extent. However, if this would be explicitly defined, breaking this would be an explicit straightforward lie, which would make it harder to make socially acceptable in some limited circles, people would have to go to greater length to hide it and this kind of thing would not happen that often.

This is a risk, yes. I don’t necessarily think it’s a big problem.

First off, you’re describing premeditated, wilfully fraudulent behaviour. I suspect a lot of the problems are either less deliberate (you react to new information by making changes to your hypothesis or protocol without stopping to think what that does to your experiment’s reliability), crimes of opportunity (“this is so close, there’s no harm in fudging it a little bit, right?”) or a combination of both. It’s ok to enact rules that fight pervasive small issues while doing nothing about rare malicious behaviour!

Even the malicious behaviour would become much harder though. First off, the strategy you described only works for setups where the data collection stage is relatively short. If it takes a long time, then the fraudster’s timelines start looking suspicious. Second, that type of behaviour could really harm you if caught.

I'm not a scientist, but in software development some people used to think that planning and designing everything up front was a good way to go about things. Turns out that taking all the big decisions before getting your hands dirty and really learning about the problem domain isn't really a great idea.

There's a long and unpleasant track record in science of people doing forming a hypothesis, doing trials, mining and changing methods and hypotheses and p-hacking for anything significant, and publishing a misleading paper that completely fails to mention that the original hypothesis was abandoned halfway through.

That may sound pretty agile. It is! You're absolutely right that this is an excellent match! The issue is that while agile may work pretty well for software dev, it perhaps doesn't work so well for science. It's given rise to a number of abuses.

It might also be noted that researchers doing experimental design are often quite familiar with their field, having gotten their hands dirty in a problem domain repeatedly.

There's the XKCD about significant P-values: https://www.xkcd.com/882/

To prevent that from happening, all the experiments that failed need to be reported too, and the link between them has to be obvious.

Exactly this kind of registration is mandatory for clinical trials under the Declaration of Helsinki and in US and EU law, but it's still poorly enforced.

If this issue matters to you, I would encourage you to support the AllTrials campaign; they have done stellar work in exposing publication bias.

http://www.alltrials.net/

This would lead to a long string of embarrassing cancelled experiments being visible in the literature, and I can only guess that's a main reason it's not done. It gets worse if the experiment was to test a controversial hypothesis (or even, bad publicity around the hypothesis being tested could shut down the experiment, or in some cases, compromise the results).

Hypotheses could be sealed upon filing. The existence of sealed hypotheses could/should be public, to prevent the system from being gamed by filing dozens of hypotheses at once for the same experiment and then selectively unsealing the one that matched their results. If the experimenter had too many sealed hypotheses to their name, that would be seen as cause for concern. But only a handful of sealed hypotheses would be unlikely to raise any eyebrows. Furthermore keeping one's sealed:unsealed ratio low would be an incentive to publish negative results. This system would also protect the experimenter from political pressure if they were testing a controversial hypothesis.

To put on a different hat for this: when I went to do my Master's thesis, I actually did not have a hypothesis nor did I have expected results. Maybe it was unique for us, but we actually did not know what to expect for the outcome of the research, that was the whole point of it.

We had procedures so that when we came up with a result, it could be validated and we made sure we didn't do anything that could have tainted the research.

Reply on news.ycombinator.com