PeerReview4All: Fair and accurate reviewer assignment in peer review
arxiv.org
arxiv.org
I would rather see some sort of prediction market where scientists make claims betting against one another, and 3rd parties are incentivized to conduct experiments based on the total value at stake.
Not all science is based on experiments. Furthermore, science is not usually done in two simple steps: creating a theory and validating it. In practice, it's a cycle of refirenement that often requires dozens of iterations and involves discovering new phenomena.
In your model, who decides if the 3rd party has invalidated the claim or simply conducted the experiment incorrectly? Everyone involved has a financial motivation, so there's no independent side to resolve this issue.
Peer review can be more structured, is already blinded in many cases, and could benefit from regular crowdsourced suggestions from a variety of scholars. In many cases, journals have begun many reform practices like valuing open sharing of materials and code, structured transparency checklists, and opportunities to publish null results. All wonderful advances that don't require hammers.
> Not all science is based on experiments.
Okay, well I'm not talking about that kind of science. I'm talking about the science where the value of a theory is measured in it's ability to make testable predictions. I'm not really sure how the the other kind of scientists resolve disagreements.
I didn't claim to have solved the problem. I was outlining what I think a solution would look like, and contrasting that to the idea of peer review as the way to determine what is scientifically relevant, if we could only make it more fair.
It has since been published: https://proceedings.mlr.press/v98/stelmakh19a.html