Comment by unlikelymordant | Hacker News Reader

Comment by unlikelymordant | Hacker News Reader

unlikelymordant

My field is signal processing (and i've published quite a few papers), but my take on what papers are is a little different. Their purpose was originally, and i think still, a way for researchers to communicate problems they have solved to other researchers. The fact that some subset of papers turns out to be wrong, or that the public doesnt understand them is immaterial. The repositories of knowledge about which papers are good, and which are bad are the research labs. Ask a professor at a research lab which papers have results that can't be reproduced, and which are high quality, and they'll know immediately.

When there are statements in the article "Such results erode confidence in academic work", the confidence is only eroded in the papers that are bad, which is how it has always worked. Confidence isn't lost in every paper, or the scientific process, but laypeople and administrators see some stat about "30%" of papers have some error, then all of must science is wrong.

Signal processing may be unusual, but it's generally not true that professors know which papers are likely to replicate or not:

https://fantasticanachronism.com/2020/09/11/whats-wrong-with...

"studies that replicate are cited at the same rate as studies that do not"

Confidence is eroded in all papers by this sort of thing, because academic structures are the same across fields. Whilst individual fields or labs may be more rigorous than others, people can't be expected to keep a massive database in their heads of which fields are good and bad. To be able to generalize about quality requires institutions that create quality, but universities don't do this. And because universities dominate "science" or at least the public discussion of it, that lack of quality control reflects on all science.

I work in signal processing and as a field of theory it presents no barriers to replication. My fiance is exiting chemistry academia due to an abusive professor. In the past they have wasted over a year of effort on one project trying to build off of prior work. The conclusion they ended up with is that the original work was fraudulent. No paper covers the inability to replicate the previous work.

So what does this mean? The professors pushing the papers out are not doing their due diligence and there is no one else doing public checking. The loop is open. It is not science. That's the crisis.

This is a fair point, but for the layperson who cannot distinguish between the 30% of papers that are bad and the remaining 70% it may as well be a denunciation of all science, since they themselves perhaps cannot tell the difference.

While papers remain the main standard by which science is communicated maybe there is a place for those in the scientific community that do know to be more vocal about which papers are worth reading.

This is something that people have to accept: the target audience of scientific papers is other researchers, not laypeople. Ideally we need more effort spent on synthesis of prior research. In my field (ML) very few prominent researchers write overview papers summarizing state of the art even over last few years. There is little academic incentive in doing so. Laypeople should read books written for them (like Hawking's books), not bleeding edge research papers.

The problem is deeper than this, though. All of the "scientific method" theory which is meant to ensure reliable results applies only to experiments and observational studies. Syntheses are just prose - there's no particular reason you should have confidence in a synthesis if you don't think the author shares your values or it contradicts something else you strongly believe is true.

I’m in electrical power engineering and control, I find the oldest papers are usually the best ones as they present the essence of a concept simply. New papers often don’t add much except a bibliography I can use to find the source and authority on a topic.

Reply on news.ycombinator.com