Science isn’t suffering from a lack of papers. It’s suffering from a lack of good papers. Making it easier to just pump out paper-mill publications is about the last thing science needs right now.
The most absurd part is that everyone in academia knows that publish or perish is tremendously damaging to real research. Yet we’re all hostage of this system that we created in the name of “merit” and “efficiency”.
We need a different system to identify and reward talented hard-working people. Back in the day it all relied on actual interpersonal interaction and subjective judgment, but there were also much fewer researchers worldwide.
This will just make research inaccessible to most researchers. There is no incentive to limit publishing, at all, other than at the highest echelons. Publish or perish will just become worse. Look at what is happening to programming and extrapolate that to research work.
And all for what? Just to keep up this facade of society until most of society can be excised, whether artificially or naturally though lack of reproduction.
In the long run conceivable we could use AI to hold papers to a much higher standard, audit all the data and code that is associated etc.
Between 2016 and 2021 the share of ML/ robotics/ AI researchers being reproducible (ie contianing code and similar instructions to reproduce) doubled [1].
The major US labs have gone largely closed source (I.e. they no longer publish frontier research) but the Chinese ecosystem has incredibly reproducible code.
This is field dependent obviously but I think it atleast gives reason to be optimistic.
Yes people will churn out fake slop research, but it feels like that can be categorized and then ignored.
> if AI can reproduce such research it would be capable of doing such research itself
Well there is a big distinction between research validation and research generation, it is generally much easier to verify that a math proof is true or false than to find a truly novel proof.
But yes in the long run I’d think AI will be doing tons of research and it will by default reproducible. So maybe we’re aligned after all?
For a while now there has been very little incentive for providing these alongside the paper, and I don't see why exactly 'AI' would change this. I could even see how making it vague to be harder to test with LLMs could be profitable for citation hackers.
The Chinese open source community has made a lot of incentive to make research reproducible for example. The most reproducible works from I.e. deepseek get widely cited and adopted.
I don’t think we can just say “AI” and it’s fixed but with deliberate effort there’s reason to be optimistic.
If it fails you may have to double check it did properly reimplement it, but if it succeeds you do get a reproduction.
repetition of materials and methods toward reproducibility, holds far less wieght than multiple variants of process designed to test a common hypothesis resulting in agreement.[null, or failure to null]
It wasn't perfect before, but it at least took some time to fake a paper. The problem is now people can produce a very plausible looking completely fake paper in minutes. Peer review is in the process of completely collapsing, in fact I think it's already basically done.
The only way this might fix things is if we require all papers are completely reproducable (that doesn't help in subjects like biology of course. They can still provide all the experimental data in the rawest format possible which doesn't break any laws).
I want to see who did the hard work properly, and who focused on publishing with concealed details.
Underlying integrity is rigor.
Underlying rigor is education.
It goes deep, for sure, IMO.