Hacker News Reader

Comment by untilHellbanned | Hacker News Reader

No thanks. Papers involving animals are already backbreakingly slow compared with cell-based or in vitro work. I know because I've been lapped by my colleagues using more simple systems as I slog through our paper we got rejected from Nature because the reviewers suggested another 3 years worth of experiments. Yep, year 5 into this single project, which we knew the outcome for 4 years ago. Not excited about this proposal at all.

Look I'm all for rigor but how about the people trying to make money off the deal pay for all the work and keep people like me out of it. Or don't allow the people trying to make money interpret the results of such preliminary studies so liberally. It's like the education system. Scientists like teachers, both of which don't make much money and do all the labor, don't want more hoops jump through.

View on HN

Sorry, which parts of the proposal are you responding to? The article is more specific than "more rigor". Are you objecting to the higher p-value threshold? The independent confirmation?

The author argues that a single higher quality confirmatory experiment will be able to replace gathering lots of statistics for exploratory experiments:

> Unlike clinical studies, most preclinical research papers describe a long chain of experiments, all incrementally building support for the same hypothesis. Such papers often include more than a dozen separate in vitro and animal experiments, with each one required to reach statistical significance. We argue that, as long as there is a final, impeccable study that confirms the hypothesis, the earlier experiments in this chain do not need to be held to the same rigid statistical standard.

Do you disagree?

> one that incorporates an independent, statistically rigorous confirmation of a researcher's central hypothesis. We call this large confirmatory study a preclinical trial. These would be more formal and rigorous than the typical preclinical testing conducted in academic labs, and would adopt many practices of a clinical trial.

As you can see above and from your quotation (and like many other folks who come in to save the day), this article is heavy on plans and short on who is going to do the work. Of course I support papers where every single experiment doesn't have to play p < 0.05 games, but other parts of the article wander in other directions. That's all I'm reacting to.

Having a higher threshold for publication can be imposed without detailing who does what work. You might argue that this means less research will get produced, but it's probably worth it. Practitioners underestimate the difficulty of transferring knowledge to outsiders because of frictions due to trust, clarity, and tacit knowledge.

http://blog.givewell.org/2016/01/19/the-importance-of-gold-s...

Eh...

When you call for everyone to scale their experiments up by sixfold, I think you also need to consider the logistics of doing that. I'm totally in favor of better, more rigorous experiments, but I know that we couldn't afford the time, space, or gear needed to do that right now.

The editorialization of publications also has a detrimental effect. There's a great PhD comic about this:

https://robertcargill.com/2012/04/21/the-science-news-cycle-...

I don't think they want this to apply to all papers, just a certain class of papers. But maybe you have already considered that(as your reviewers obviously are already pinging you about more testing) and you still disagree?

Reply on news.ycombinator.com