Surely those seemingly smart anonymous reviewers now feel pretty dumb in hindsight.
Peer review does not work for new ideas, because no one ever has the time or bandwidth to spend hours upon hours upon hours trying to understand new things.
Peer review does not work for new ideas, because no one ever has the time or bandwidth to spend hours upon hours upon hours trying to understand new things.
There's an article I came across awhile back, that I can't easily find now, that basically mapped out the history of our current peer review system. Peer review as we know it today was largely born in the 70s and a response to several funding crises in academia. Peer review was a strategy to make research appear more credible.
The most damning critique of peer-review of course is that it completely failed to stop (and arguably aided) the reproducibility crisis. We have an academic system where the prime motivation is the secure funding through the image of credibility, which from first principles is a recipe for wide spread fraud.
Academic careers are then decided by the Github activity charts.
But is there an alternative that still allows most academic aspirants to participate?
My point was that academics could use Github (or something like it)
It should be possible to use something like Github to *verify* "that the code compiles and 95% of test cases pass" instead of just "pinky swearing".
Maybe something like nbdev.fast.ai.
In any case, it was just a thought, and likely not an original one. I would welcome it if someone tried to build this and proved it can’t be done.
Thank you for the stimulating discussion!
What is the actual line of argument that demonstrates this success/usefulness/etc... can be reproduced in your envisioned system?
> I would welcome it if someone tried to build this and proved it can’t be done.
It's impossible to prove a negative, so this doesn't make sense. Did you mistype?
mostly it should try to do it through falsifying things, of course groupthing is seldom effective at that.
It's worth pointing out that most of everything happened before peer review was dominant. Given how many advances we've made in the past 50 years, so I'm not super sure everyone would agree with your statement. If they did, they'd probably also agree that most of the worst science also happened before peer review was dominant, too, though.
This seems unlikely to be true, simply given the growth. If you are arguing that the SNR ratio was better, that's different.
https://www.experimental-history.com/p/the-rise-and-fall-of-... https://www.experimental-history.com/p/the-dance-of-the-nake...
https://www.experimental-history.com/p/the-rise-and-fall-of-...
- accessing and verifying the datasets (in some tamper-proof mechanism that has an audit trail). Ditto the code. This would have detected the Francesca Gino and Dan Ariely alleged frauds, and many others. It's much easier in domains like behavioral psychology where the dataset size is spreadsheets << 1Mb instead of Gb or Tb.
- picking a selective sample of papers to check reproducibility on; you can't verify all submissions, but you sure could verify most accepted papers, also the top-1000 most cited new papers each year in each field, etc. This would prevent the worst excesses.
PS a superb overview video [0] by Pete Judo "6 Ways Scientists Fake Their Data" (p-hacking, data peeking, variable manipulation, hypothesis-shopping and selectively choosing the sample, selective reporting, also questionable outlier treatment). Based on article [1]. Also as Judo frequently remarks, there should be much more formal incentive for publishing replication studies and negative results.
[0]: https://www.youtube.com/watch?v=6uqDhQxhmDg
[1]: "Statisics by Jim: What is P Hacking: Methods & Best Practices" https://statisticsbyjim.com/hypothesis-testing/p-hacking/
For example just see this from the review of f5bf:
"The main contribution of the paper comprises two new NLM architectures that facilitate training on massive data sets. The first model, CBOW, is essentially a standard feed-forward NLM without the intermediate projection layer (but with weight sharing + averaging before applying the non-linearity in the hidden layer). The second model, skip-gram, comprises a collection of simple feed-forward nets that predict the presence of a preceding or succeeding word from the current word. The models are trained on a massive Google News corpus, and tested on a semantic and syntactic question-answering task. The results of these experiments look promising.
...
(2) The description of the models that are developed is very minimal, making it hard to determine how different they are from, e.g., the models presented in [15]. It would be very helpful if the authors included some graphical representations and/or more mathematical details of their models. Given that the authors still almost have one page left, and that they use a lot of space for the (frankly, somewhat superfluous) equations for the number of parameters of each model, this should not be a problem."
These reviews in turn led to significant (though apparently not significant enough) modifications to the paper (https://openreview.net/forum?id=idpCdOWtqXd60¬eId=C8Vn84f...). These were some quality reviews and the paper benefited from going this review process, IMHO.
For example, a reviewer essentially insisting that nothing is worth publishing if it doesn't include a new architecture idea and SOTA results... God forbid we better understand and simplify the tools that already exist!
I'm not sure peer review makes things more rigorous, but it surely makes it more slow.