Hacker News Reader

Comment by zer00eyz | Hacker News Reader

The review process is broken.

Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar.

Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from.

The current system is glad-handing, intellectual protectionism and mastrubation.

Academia has only itself to blame for this, and they are apparently unwilling to fix it.

View on HN

"Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from."

This is already how it's supposed to work. The review before publication is a fairly superficial check that just confirms that what you describe follows basic scientific practices. There is no validation of the actual research. A proper reproduction is what's supposed to come after publication.

IMO the real problems are that a) there isn't much glamour and funding for reproducing other's studies and b) "science journalists", university PR departments and now in part people on social media are picking up research before people on the field looked at it or misrepresent it. Suddenly the audience is a lot of folks who never were the intended audience of the process.

There is a pretty simple way to change all of that.

Academic standards: You are not longer allowed to site a non reproduced paper in yours.

Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.

> You are not longer allowed to site a non reproduced paper in yours.

I fully agree that in an ideal world, that would be the case. But some reproductions (especially now with machine learning) could cost millions of dollars and years to do. I don't think that's a reasonable or feasible thing to require.

Well, then, at least there should be pressure around having to mention this : a non-replicated study (by an independent group) is after all inherently suspect.

>> could cost millions of dollars and years to do

https://en.wikipedia.org/wiki/List_of_colleges_and_universit...

Not only do they have the money they are charging the students more than ever.

This is a very fair criticism and something that absolutely should be discussed, although I think it's a separate issue as papers can be written and published by anybody, regardless of backing from academic institutions.

> Academic standards: You are not longer allowed to site a non reproduced paper in yours.

> Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.

The reproducibility crisis is just a symptom of the publish-or-perish culture that pushes academics to churning out bad research. Academia already over-emphasizes publishing positive results at the expense of studying important questions. Your solution would further incentivize low risk, low impact research that we have too much of.

Aside from that, there are a lot of edge cases that would make this difficult. If I do five studies that are modifications of each other, and all show the same basic effect, but I publish it as one paper, does that count as being reproduced? What if a result has only ever been reproduced within a single research group? Does the Higgs Boson need to be confirmed at at a collider outside the LHC?

>> What if a result has only ever been reproduced within a single research group? Does the Higgs Boson need to be confirmed at at a collider outside the LHC?

Yes. It does. Take out all the detectors and let another team build their own and come in and prove it. The really expensive part was the big concrete doughnut in the ground. After that wee could ... you know... shut the fuckin thing down so we stop pouring money into a literal hole in the ground. Idk we could do some more science after that. What is missing from quantum theory that the LHC running is going to find?

IF we stopped funding mastrubatory string theory and put more of that into practical physics maybe we would have had something else for the LHC to do...

>>> Academia already over-emphasizes publishing positive results at the expense of studying important questions

Academia is made up of Academics, they aren't inclined to fix their own problem... it as effective as the church policing its own issues.

I love what Jack Horner has done (see: https://creation.com/dino-puberty-blues ) but he would never have been able to achieve any of that if those people were alive and his peers.

I think we need to find ways of giving status to reproducing studies. Maybe not as much as novelty, but definitely something greater than what it is currently.

IMO reproducing findings could/should be a mandatory part of PhD training.

Reply on news.ycombinator.com