Here is an argument that peer review is basically a failed experiment: https://experimentalhistory.substack.com/p/the-rise-and-fall...
Peer-review doesn't catch fraud and is sometimes a political process. I've found the best corrections I've gotten is after posting pre-prints to online forums. I suggest that commenting on pre-prints is better than peer-review pre-publication. I imagine a ranking system could highlight comments from trusted reviewers. Studies that no one wants to review were probably never going to be read anyway, and so there was never any reason to review these studies anyway.
Any peer review added to that might improve some things, but is it worth it to add ~6 months to the publication timeline for a marginally improved manuscript? I'd say no.
Science did just fine before it, as it will after it's phased out.
Look at stable diffusion as an example. Incredible papers, such as dreambooth, LORA, controlnet, are:
1. Published on arxiv before peer review
2. Productionised within 2 weeks of paper release (peer review not needed)
3. Community rapidly adapts tool, makes it easier to use.
4. Products built on such papers proliferate extremely rapidly within another few weeks.
In this system, peer reviews are worthless. The github code quickly demonstrates whether a technique is useful or not, and the community adoption rates replace citations as proof of a paper's power.
This is why AI art can progress at such insane rates, weeks from paper release to widespread productionisation.
Obviously, this won't work in most other domains, because there's no equivalent to mass consumer interest, open source communities, and low-cost experiments. But it does represent the ideal of an academic research paradigm.
This is the key to everything else. There is built in reproducibility and amplification of new, functional ideas in the ML community.
For the most part in life sciences, papers are published to achieve current grant aims and write future grants that will be funded. You can be an academic and love your research area and be ultra-passionate about it, but at the end of the day, grants are the end product that you are working for.
Your science does not have to work or be replicated, all you need to do is publish papers that make grant reviewers think you are reliable enough to not waste federal grant money. Nobody on the grant review board has time to look carefully to see if you papers are not fraudulent.
Let’s look at physics on the early 20th century, which had progress even faster than today’s machine learning research. Massive upheavals and rapid progress in understanding our world, including 4 different models of the atom (including the most correct one we still use today) and general relativity. What’s the difference to today’s life sciences? At the important epicenters of the day, working in the field was 1) contributing new observations, 2) directly testing somebody else’s theories with an experiment.
In today’s world, very rarely will somebody contribute new observations without an underlying motivation (get new grant money, advance current grant claims). And nobody has the time or resources to test other people’s ideas with new experiments. Why? Cause research is expensive and you would need a grant to fund a replication. And no government body funds those grants.
Disclaimer: there’s people in life sciences in some fields doing good work.
Pre-registered trials and/or arxiv + open science.
My personal philosophy about such things is to think of Hanlon's razor as a boundary condition. The longer an institution has been around, the more likely the incompetence is actually just well disguised malice.
A research paper is supposed to be a honest report of best efforts to study a topic. If it's not, that's a problem that can't be solved by having a few people spending a few hours with the report. The paper is not the final word on anything anyway. If you read a paper expecting to learn something about the world, you are doing something wrong.
As a reviewer, you determine whether the paper is interesting and relevant to the venue. You report any issues you spot that should be corrected and any things you believe that could be improved. And if you get any ideas you feel like sharing, you may share them as well. And that's it.
Can you expand on this? As a layperson, I'm wondering what would you read a paper for at all if not to learn something about the world? What's the point otherwise?
Published results are often contradictory, because individual papers are unreliable. Something may have gone subtly wrong, the interpretation could lack nuances, some key understanding may still be missing, or the authors may have just been unlucky. When an expert reads many papers on related topics, the arguments shape their beliefs. Eventually a scientific consensus may emerge, which is the next (but still an insufficient) step towards reliable knowledge.
https://twitter.com/TheSavageInMan/status/108350367715796172...
I think people get in trouble when they think academic publication has value in itself. Ideas become valuable when people care about them. Publication is one particular path to reach a community of readers but it doesn’t make your ideas matter.
But if something has predictive power, it's valuable no matter how many other people know it. In some cases, it's way more valuable when it is still unknown, because nobody has had the chance to capitalize on it yet.
In some instances, it won't end up being useful because it's only relevant to someone in a lab full of million dollar equipment. But that'll only be the case sometimes.
My approach is to avoid framing things in terms of "now I know x is true", and instead look at it in terms of "someone believes (or wants me to believe) x is true". I then weigh all of those beliefs as I observe the world around me. I resign myself to never really knowing anything myself, but having some idea of what different schools of thought are on a given topic.
If you move your information gathering further along the chain, to maybe a text book, or some expert's twitter feed, why should these things be more reliable? If they've correctly come upon the consensus view, you still have to consider that informing you of the truth night not be their first priority. They could be after money, advancement in their field, political agendas, and (though less likely) they still could be just plain wrong.
So they're more likely to know the truth, but you're still unable to evaluate whether they're giving it to you or not.
I beg to differ. Peer review is an adversarial process. The author(s) of the paper are making statements and report findings proving them by logic or/and data and analysis. The priors while reviewing a science paper is "wrong" until proven "correct". This also covers accuracy and veracity of the data and analysis. The only thing a peer review is not is assigning intent or blame. It is not the job of a reviewer to look for fraud when simple incompetence could explain it. But after reading this article I will add fraud dimension to a list of fallacies I am looking for while peer-reviewing a manuscript.
1. Validity of the experiment
2. Interestingness / novelty of the work
3. Appropriate choice of methods and correctness / believability of the results
4. Signs of outright fraud (suspicious figures, etc.)
Absolutely if I notice some weird Photoshop artifacts in a figure, or some other obvious sign of fake data, I'm going to call that out. I'm probably an outlier on this next one, but I've even been tempted to reject articles just for having egregiously, unreadably bad writing. I know this will be regarded as bias against people whose first language is not English, but if the writing is so bad that it stands in the way of making sense of the article, and the authors can't be bothered to get a decent editor, it's not a worthy contribution to the (English language) scientific literature.
The problem people have conceptualizing peer review is that reviewers can't reliably spot stuff; they simply don't have time to do it, and it isn't the premise of the exercise.