Hacker News Reader

Comment by freediddy | Hacker News Reader

No paper should ever be referenced until it has been replicated by an independent group. The replication crisis in science is real, and it won't take long for no one to believe anything anymore.

View on HN

Not all science has this issue. But to reference Thiel, a lot of the sciences with "science" in the name, seem to.

That rule would make it difficult to report on a failure to replicate a published paper.

Every paper needs to include an independent replication of the study.

If only we had that level of science funding.

So build a second particle collider to validate what comes out of CERN?

Does some sort of batched style of research already exist? By batched, I mean a study where n number of teams all agree on a single methodology, and then conduct the same experiment simultaneously. Each team could then publish their results independently and the teams could publish one aggregate.

I know meta studies attempt to do this in a way, but they can sometimes have their own problems too.

I cannot recall coming across this style of research in much of what I have read as a hobby, but maybe it is more common in some domains than others? I am not smart, and there is nothing new under the Sun, so surely something like this exists, no?

Not quite what you're asking, but pre-registration does alleviate e.g. p-hacking and such without the need for multiple teams. It doesn't do much for outright fraud, but still...

Only if the study matches its pre-registration and someone is checking that. Which it turns out, they often don't.

https://www.compare-trials.org/

That's a fair point. Ultimately, incentives are all screwy at the moment.

Gotta tie some actual prestige/shame to not following up.

It doesn’t seem very popular but that exact thing does seem to exist, here called a ‘Registered Replication Report’:

https://www.psychologicalscience.org/publications/replicatio...

There are some completed which is cool. I agree that this seems like a great initiative.

If the failure is replicated by multiple groups, that would be reportable.

Otoh, not sure that anyone wants to read a report that some finding that nobody reported on couldn't be replicated.

>The replication crisis in science is real

In psychology, you mean. Many contest if it even qualifies as a science anymore after the complete annihilation of trust it has undergone.

In the physical sciences, any sufficiently important paper is independently verified. Sounds like cost/effort of these multi year studies is not faesible to replicate, but then again, I know nothing about social science so w/e

> The replication crisis in science is real, and it won't take long for no one to believe anything anymore.

If you think replication in the sciences is in crises, wait till you see the field of psychology.

Reply on news.ycombinator.com