Comment by SubiculumCode | Hacker News Reader

Comment by SubiculumCode | Hacker News Reader

"The most common reasons they gave were that the studies didn’t achieve statistical significance, a controversial but commonly used threshold for publication; that the data were part of a pilot project; and that there were technical issues with the animal models. But none of these is a valid excuse to not publish your findings in the scientific record, says study co-author Kimberley Wever, a metascientist at Radboud University Medical Center. “All animal studies should be published, and all studies are valuable for the research community.”

I hard disagree that pilot projects and results from data with technical issues should be published. There is a real cost to write-ups (time). Pilot projects usually are done just to prove feasibility of a protocol, and data retrieved from experiments with technical issues is likely noise. This could include inaccurate measurements, cross-contamination, etc. Taking the time to write and publish all the junk is, frankly prohibitive. I do think null-results should be publishable, however.

The title is a bit of click bait also: The animals are not missing. Make a request of the relevant Institutional Review Board.

Are any of you voting my comment down actually scientists?

If you have time to do the work to get a pilot project off the ground then you absolutely have time to do a short write up on how it failed.

Pilot projects are usually a small sample that demonstrated that a protocol can be achieved, not a meaningful set of data to test a hypothesis. So no, it is probably a waste of time.

Whether or not a protocol can be achieved using a certain method is a valuable outcome. Write it up so your peers know what didn’t work.

Yes. The details of the used protocols are supposed to be included (and usually are) in the primary publication of the research.

But not the results of the pilot unless it was promising?

pilot data is not what you think it is. Pilot data is about finding parameters, testing procedures. It is neither a consistent nor a full replica of an actual experiment.

I think the thing that non-experimentalists don't realize is how many experiments fail for reasons that are not understood. For example - "the neurons in the part of the brain failed to express the transgene I wanted to use to study them." To actually write a paper about that null result requires substantially more experiments to form hypotheses and test them. Moreover, my lab is far from being an expert in the esoteric ways that this can happen, so when I try to submit the paper, the experts ask for even 10x more experiments. So I've gone farther and farther from the questions that are "important" (reflect my interests, expertise, and funding).

But this explains the value of scientific meetings and poster sessions where I can just tell this random tidbit to interested people without the burden of peer review.

An interesting recent development I've observed once was a tweet describing the fact that a dye (Texas red) labels brain vasculature even when injected subcutaneously. This random fact is not in the literature but is quite helpful from a procedural perspective. I think that science twitter has a potentially super valuable role to play in reporting unexpected or otherwise difficult to publish findings.

I think that is a fair point. It is partially answered by whether a grant was received, but only partially.

On the other hand, data sharing protocols mean the data is going to be made available, so I think that probably addresses that issue.

The problem with pilot data is that lots of things can change in the course of running the pilot; tweaks of the experimental protocol, bug fixes to code, etc.

Most "failed" pilots don't demonstrate that an approach is fundamentally and irredeemably flawed, which could be an interesting paper.

Instead, they demonstrate that doing something this specific way is too much of a hassle, too unreliable, or too expensive to answer a particular question in a particular context: skills, budget, timeline, resources, alternative ideas, etc.

I don't see how you could write pilot results up ("Trust me, I'm usually pretty good at these sorts of things but this didn't work well"?). Meanwhile, disentangling these factors in a rigorous, generalizable way would turn it into an entirely different project.

Actual scientist. I believe Record keeping should be kept, i'm not saying it justifies a full publication but these records should be kept. If you go on to publish then these records should be acknowledged in the publication. It can be included in appendices / extra information even if it's not in the main article.

Reply on news.ycombinator.com