Hacker News Reader

Comment by Jabbles | Hacker News Reader

That's not going to work for Astronomy, or results of multi-year studies.

Long-term studies or not, it would be useful to have an up-to-date system that clearly indicated whether or not the results of some particular study had been replicated, and how many times they had been replicated if at all, and if there were any conflicting results between studies.

That is what Cochrane does in the UK. And indeed it is great.

it will, if you require that the data is shared and can be analyzed by another group. In some cases it will slow down the progress initially, but I think in the long term we would be getting much better results.

You can fabricate data nearly as easily as you can fabricate results. It is easier to detect fabricated results from raw data than it is from the final results, but fraud adapts to the methods used to detect it. If you create a new way to detect fraud, fraud will change to avoid that method. Detecting fraud is an arms race.

At some point it becomes easier to just run the damned experiment. Similar to buying a safe based on making its contents not worth the trouble to crack it.

This would be a good first step. "Photoshop for biologists" (as a workshop) is very much a thing.

You're already moving your goalposts. Your first post said replicates the finding. If I simply post my results and someone else looks at the data and agrees with me then are they really replicating my finding or did they, oh what's the word… peer review my work and agreed?

I really dislike the idea that "Your code runs on your data" is a form of replication.

It's a minimum standard and one that many published works cannot meet. Universal enforcement of this minimum would itself be a big step forward.

It is a minimum standard, and a useful one. It mostly annoys me when certain groups of advocates seem to treat it as an end in and of itself, and evidence that something is "repeatable" and will end the problems in obtaining scientific evidence.

I come from a field where something isn't a reproduced until it's also found in an entirely different study on a different population.

Oh, same here. Two of my current manuscripts have now been replicated in 3-5 independent clinical cohorts. The thing is, if the field won't even recognize that "ability to produce the same outputs given your inputs" is critical, good fucking luck with things like "actually replicates in a separate population".

Neuroscience is particularly horrendous about this.

Reply on news.ycombinator.com