Hacker News Reader

Comment by a-dub | Hacker News Reader

i've said it before and i'll say it again, this isn't going to improve until there are more grant dollars and recognition for replication than there is for novel discovery.

replication is boring, yet critical, and we should be compensating highly skilled scientists for doing it.

View on HN

It won't change until a track record of producing studies that fail to replicate hurts the career of an academic more than not publishing anything at all. As long as publish-or-perish provides a stronger incentive than the negative repercussions of publishing nonsense, this will continue.

I don't know if one necessarily wants to punish for failure to replicate, but I do think that falsified data/records should lead to a death sentence of a scientific career or result in criminal charges if necessary (e.g. for defrauding the associated institutions/grantor). I guess sort of like what Elizabeth Holmes is facing currently.

That sounds like spending money on trying to replicate results that can’t be replicated doesn’t it? Isn’t that what this study was doing? Doesn’t it make more sense to revisit how the existing funding is spent?

I have very little insight into this area of research, so I’d just have to guess everyone’s in a hurry and reaching for results or published papers.

> Doesn’t it make more sense to revisit how the existing funding is spent?

i don't know what you'd find in a financial audit that a scientific audit wouldn't tell you, but a passing financial audit won't tell you if a scientific finding actually sticks. therefore, seems like a better deal to audit the results, no?

I didn't mean spent in a financial sense. I meant in how the research itself is performed (or maybe how it's granted and re-upped), as that seems to be the issue. If research continues as it is, won't most reproduction studies still fail if this posted study is accurate? It makes more sense to treat the cause and not the symptom.

Part of this is being addressed through increased reporting requirements for data, software, analysis pipelines, pre-registered protocols, and the like. This has led anecedotally to some improvements in quality, and at minimum it makes initial due diligence on the publication significantly more impactful.

> "i don't know what you'd find in a financial audit that a scientific audit wouldn't tell you"

Financial audits would uncover financial fraud.

> "seems like a better deal to audit the results, no?"

Better to audit both.

i suppose. i suspect most financial fraud would be caught by the accountants and grant administrators at respective institutions. accounting and best practices around it have been around for a long time.

this discussion is about ensuring scientific velocity by increasing the rigor required to declare success and add new knowledge to the commons.

unless there's large scale fraud going on, i honestly don't think bringing in additional auditors to look at books that are already professionally kept is going to benefit anyone other than said auditors.

> If it was only their money on the line, that would probably suffice. But when researchers (and by extension the institutions they work at) are receiving government grants, I think there need to be independent audits.

these are all nonprofit institutions, some are government run, i'm pretty sure they're required to engage independent professional auditing firms just as a condition of their tax exempt status, let alone acceptance of federal grant dollars.

> i suspect most financial fraud would be caught by the accountants and grant administrators at respective institutions.

If it was only their money on the line, that would probably suffice. But when researchers (and by extension the institutions they work at) are receiving government grants, I think there need to be independent audits.

100% agree. The NIH could replicating studies a requirement of funding for graduate or postdoctoral training. In addition to the scientific and societal benefit, it would just be good training.

I would be glad to have funding explicitly set towards appropriate and open documentation of the actual protocols, equipment, and materials, plus a combined effort to enhance these reporting requirements by the journal. There has to be a balance between the one-off interesting new findings and the large-scale work, but clinical trials definitely fall under the latter category.

your first task in lab will be to reproduce our competitor's work!

in seriousness though, there's enough money and trained scientists looking for work... moreover i think there's a whole personality type that is prevalent in science that would be very happy to replicate and reproduce results...

This is a self-resolving problem.

- breakthrough study is published

- because it’s breakthrough, it attracts attention and other labs look at it (we often tried to repeat breakthrough work in our lab)

- if it can’t be replicated, interest just peters out OR it drives someone else to “fix” it. If it’s so important people need to know it gets published.

- if it can be replicated further research continues

The problem is with step 3: incredible amount of energy goes into building upon a bad study because the fact that it's not retracted.

95% of science is doing stuff that doesn’t work. Replication - independent replication - is not a wasted effort.

Reply on news.ycombinator.com