Hacker News Reader

Comment by lp4v4n | Hacker News Reader

I once read the criticism that science is already moving too fast and that wrong results get published and accepted as true because nobody reaps any laurels for verifying or reproducing results, even if that's an indispensable part of the scientific method.

Now in the age of AI this problem will be heavily accentuated. Who's going to review the quality and validity of dozens and dozens of papers being generated by a researcher who used to publish sporadically?

It's not hard to see science going through a "slop crisis" in the next few years, in the same way that now projects are getting inundated with PRs.

View on HN

Yeah, science needs to reinvent itself.

How about this: instead of going through publishers, researchers are required (by their funding organizations) to put a bounty on falsifying the result (for, say, 10 years). The higher the bounty, the higher the 'impact factor'. Put your money where your mouth is. A bounty of at least the cost of the study should be expected for serious research.

I can imagine a 'falsification insure' industry. They'd look at reputation as well as employ people close to the subject matter to asses risk, taking over the role of peer review. But with aligned incentives.

This way researchers and institutions who don't care too much about truth are marginalized; they either have low impact factors or they'll need to pay huge falsification insure fees.

And researchers who are sceptical of certain results will have an incentive (money!) to try to replicate it.

Remaining question: who is the authority on whether a publication has been falsified?

> nobody reaps any laurels for verifying or reproducing results

That's not entirely true. There's a whole genre of "response" and "response to response" papers that voice concerns when there's a methodological issue. That's somewhat common in my fields (computational biology and biomedical research).

And even if a "response" isn't published, people still try to reproduce published techniques if those seem useful.

A lot of useful studies take weeks, months, or years to do the first time, and require a staff, a grant, etc. at least in medicine, physics, chemistry...

You'd be surprised how rarely (overall) reproduction / verification is attempted. Sadly, a lot of research is taken at face value. That's done quite often with AI/tech stuff these days, and even more so for scientific research. It's a hard problem.

> You'd be surprised how rarely (overall) reproduction / verification is attempted.

I didn't say it's common, I said that stating it's "never" reproduced is false.

Which is interesting because the solution should be obvious: use AI to reproduce results and test what's published, instead of using bogus AI to publish more bogus results.

I agree, will be interesting to see the evolution. I know notable sources from the US academia that points to the problem since way before the AI, even if the they are from human science and economics, don't know much about the STEM status. Accordingly, the first slop crisis got a big hit with the initial woke wave in the mid 2000', then another hit during COVID, in which thousands of pointless papers about the effect where produced and an immense amount of research effort was towards it, and now AI. Science will keep progressing, even at a faster piece, the real problem will be to justify the work and output of thousand of people and researcher that currently need a salary. It may be that the academia will adjust to split between professionals focused on the pure teaching skill, while the research and publication aspect of academia will go to have much higher barriers and restricted to less people. Relevant publication will not become a standard requirement for a successful academic career but also teaching skills.

Reply on news.ycombinator.com