Comment by beloch | Hacker News Reader

Comment by beloch | Hacker News Reader

"Confirmatory labs would be less dependent on positive results than the original researchers, a situation that should promote the publication of null and negative results. They would be rewarded by authorship on published papers, service fees, or both. They would also be more motivated to build a reputation for quality and competence than to achieve a particular finding."

Sounds great, but how would this actually work. Nobody is going to get juicy grants from existing funding agencies for being a "confirmatory" lab. Nature sure as hell isn't going to pay for this. Most researchers probably can't afford to pay an outside lab to duplicate their research. Is Nature going to suddenly start refusing papers whose results haven't been reproduced elsewhere? That's basically suicide for their journal because researchers are frequently in a race with other researchers to publish first, so why publish with a journal that requires you to double your budget to pay a confirmatory lab and wait months or years for them to do the job? The pressure will be intense to publish elsewhere first.

I have a simpler solution.

Don't just slap the names of confirmatory lab authors onto other papers. Publish original papers and publish confirmatory papers with equal prominence to the original papers. Hell, devote a portion of Nature to doing just that. Currently, if you want to publish a paper about confirming someone else's original findings, not even a third rate journal will touch it unless you put at least some kind of novel-sounding spin on it. Nature should use all that scummy impact factor gaming they do to make confirmatory papers respectable. Only when the work of reproducing results gains labs respect will funding agencies start supporting "confirmation labs". At present, such "unoriginal", "hack" work is not respected at all, and Nature is a big part of the reason why.

> Most researchers probably can't afford to pay an outside lab to duplicate their research.

Even if they could, we probably don't want the researchers paying for their results to be duplicated. This would create perverse incentives, similar to what happened with investment banks and credit rating agencies. If the original researchers must get their results confirmed in order to get published, and it is them who are paying for the confirmation, they will naturally tend to choose confirmatory labs that are more likely to confirm their findings. Since the labs would then rely on the researchers for funding, that would create pressure on the confirmatory labs to adapt their methodologies in ways that make it more likely that results get confirmed (even when the original study may not warrant it).

We want confirmatory labs to have no special interest in either confirming or disproving a particular study, but in improving the overall quality of research.

Since a journal's reputation depends (at least in part) on the quality of research it publishes, the journals would seem to be the natural candidates for the source of funding of confirmatory labs. Whether they'd actually be willing to do it another matter...

Any confirmatory lab would have to be licensed in order to get grant money for it. Just like CPAs who do an audit. Sure there is some corruption and drift toward hiring more lenient firms, but it basically works.

Side note: it is weird to me that everyone talks about whether researchers can afford to pay for confirmation, but researchers never pay for anything, grants pay for everything. The granting institutions might even be excited to try a confirmation process.

That won't get you a PhD though. All PhD programs usually require original research. And the PhD students are usually the ones doing those experiments.

You're right, this would really alter the dynamics of a PhD. As is, it takes ~2-3 years since the end of experiments and beginning of writing of an article until the that article is actually released and printed. Granted, this varies by field a tremendous amount, but lets just call it ~2.5 years of 'publication hell' on the average for an article. If you were to add into this the need for confirmation, you are adding in not only the time it takes to re-run the experiments, but also the time it takes to draft them up and go through another shorter round of 'publication hell'. Lets call that 6 more months of experiments and 6 more months of publication hell. Then you have ~3.5 years of sitting on the data and the experiments since you last touched the bench with that experiment.

Many PhD programs have requirements for the number of first-author papers you must publish to get your degree. I know of one that requires 3 first-author papers to graduate, granted a little high. Running with these very loosey-goosey numbers: Say your experiments work well and only take 6 months to preform. That's then a total of 4 years just for the first paper to get out, no sooner. As you are working on the drafts, you also preform another experiment immediately after finishing the last one , so lets call it 9 months for the added time it takes to get the last one typed up. Now we are at 1 year and 3 months with 2 successful experiments and some progress of getting the first experiment into publication. Lets do that again, and you are at 2 full years and some progress on your first and second papers. That means at a minimum, you then can get out of grad school, with the new confirmation studies to do, at ~5.5 years out. That is not too bad, right?

Now what happens when that first study was not repeatable and you have to start over? That means you have to do another experiment starting the ~3.5 year process all over in your 4th year. Now we are at 7.5 year total. What happens when the last experiment is non-repeatable? You have to do another experiment starting at year 5.5, for at total of then 9 years in grad school minimum. And yes, you can be trying for 4th and 5th papers this whole time, lessening the time in school, but each also has some chance of failure.

And this is only if you are running experiments all day and they only take 6 months total to do successfully, even the repeats and new studies that don't fail you more than once; the re-do of the last one is successful and also does not fail.

Yes, these number are a bit out-of-the-hat, but I think we can all see that the length of stay in grad school either must increase by a lot, or the required number of papers must decease by a lot, if they are to be required at all. Either way, you are not getting much education while some lab elsewhere (maybe your competitors) is sitting there trying to replicate your experiments with some unknown amount of success or efforts to do so and you are just waiting and praying that you don't get screwed into a decade of grad school.

I think the best balance would be something like:

"You must include a confirmatory study by an independent lab in order to publish this research, BUT we will give you an accept/reject decision on your paper prior to doing that study."

That way, there's much reduced bias in the confirmatory results since the paper gets published either way. And if the paper would get rejected even if the confirmation is successful, then that's a ton of wasted effort and money that are major disincentives to trying at all.

As it should be, science is about experiments and predictive results, not outcomes being "desirable". Let's incentivise that.

I think there's a broader problem though -- the rigid coupling between a) whether you've published, and b) whether you've done something worthwhile, and c) the minimal size of a publishable unit.

This forces scientists to "repurpose" the journal article to signal a lot more than "hey, we did this and something interesting is going on." Anyone who contributed wants to get their name on it, and the publication is viewed as some reward rather than a test of an ideas merit (as in the "we promise to publish even you're wrong", that you're describing).

I think it would be better if we had some fine-grained system for documenting everyone's contribution, like a git DAG. Then you could separate the question of "Alice collected this data" from "Bob proposed a great hypothesis about it" and "Charlie tried to replicate it" and "so-and-so has something publishable".

> Most researchers probably can't afford to pay an outside lab to duplicate their research.

This cost would be built into the grant that funds the lab, and I bet everyone would get used to it after a few years of complaining.

anonymous messages are sometimes very powerful, measurements that are easy to make are not necessarily the right ones to make

sometimes you get snowed in, often you get snowed under

Reply on news.ycombinator.com