Hacker News Reader

Comment by dbrans | Hacker News Reader

Replication is unpaid QA on someone else's product. Nobody gets tenure for finding the bug, hence the field has been meaning to get around to it for 20 years.

View on HN

And the incentives are arguably worse than QA: you get little credit if the result replicates, and potentially a lot of grief if it doesn't. Novel results build careers. Checking them mostly doesn't

On the other hand, this seems like a super interesting path for an (unglamorous) PhD -- replicate the N most important findings in a field, subject to some restrictions (maybe focus on the ones that are "easy", or haven't been replicated yet, or the ones that could be patched together to make an acceptable PhD thesis). This sort of stuff really appeals to me, I think I might end up doing something like that in 5-10 years.

Good luck finding an advisor/a department that awards PhDs for that.

“It’s not novel”.

I think you're right that this will be the most common objection, but wrong that it will be exceedingly difficult to find an advisor that would support this. I assume that if you _can_ overturn some foundational result, you won't face criticism over novelty (potentially you can argue that overturning important accepted results is inherently novel). Then the rest is just math:

Assume that n% of foundational studies are demonstrably incorrect if replication were attempted. Then if you replicate X foundational studies, there's a 1 - (1-n/100)^X chance that you'll show one foundational study to be incorrect. Then I think you just need to find an advisor who believes the incorrectness rate n, and the replication attempts X is high enough for you to revert a foundational study over the course of your PhD.

I share your enthusiasm for this. Creating a map of the most influential research across all the fields to identify the highest priority work, any work replicating that work or validating that work by building from it would be a good start. Ideally funded by governments, but institutions, companies, and individuals could put up bounties for replicating particular research.

Universities, institutions and publications who care about their reputation should pay people to replicate studies performed by people they employ in their name. Otherwise they will loose their trustworthiness.

But perhaps it is not as big a shame for them as some think it should be...

Reply on news.ycombinator.com