Hacker News Reader

Comment by jltsiren | Hacker News Reader

How do you determine that a mistake is serious enough that it actually invalidates the paper? How do you know that it's not possible to correct it and reach a similar conclusion with fundamentally the same approach? Especially when the fix would be complex enough to justify a follow-up paper.

View on HN

You can get a good idea by reading Retraction Watch and following a few papers/scientists who show up in it. That defines the current norms.

To me, a single significant error of any kind brings the entire paper into question. If a less important figure contains an image duplication, that makes me wonder if I can trust any of the images.

That's the administrative aspect of the question. But as a scientist, I'm more interested in knowing if the result is real. Sometimes even fatally flawed papers make valuable contributions, because the proposed approach will lead to the claimed result, once done properly.

I'm not aware of any fatally flawed paper that leads to the claimed result when done properly. Could you share an example?

It depends on what you see as a fatal flaw, and if you count mathematics and theoretical fields or just actual science. In mathematics and theoretical computer science, it's pretty common that major results are initially wrong, and fixing the errors can take a long time. Wiles's proof of Fermat's Last Theorem is the best-known example.

In any case, my point was that rejecting results due to technical flaws is intellectually lazy. As a scientist, your job is more about trying to find value in other people's work than finding excuses to reject it.

I can't speak to math and theoretical CS, because both of those fields now have methods to make proofs that can be verified (and math is probably the only situation where you can "prove" something true; all other fields are effectively probabilistic, not logical in nature).

I disagree with your premise. Part of our job as scientists (thankfully no longer mine) is to reduce the irrelevant and incorrect noisy as early as possible. I have seen so many grad students get excited by a paper and put enormous effort into reproducing somethign that was a false or fake result.

As scientists, we need to have effective filters to quickly ignore things that are probably not relevant to us. But we also need to acknowledge that those filters are necessarily noisy, and they don't tell much of the value of the work they reject.

If you want to determine reliably whether something is irrelevant and incorrect noise, determining whether there is anything of value is a necessary first step.

I've seen many reproduction attempts in bioinformatics fail, because they person trying to reproduce the work didn't have the conceptual background to do it correctly. Instead of spending enough time studying the theory, they rushed directly to action.

Haha, Sean Eddy used to complain about reproduction efforts in bioinformatics (specifically, people trying to benchmark HMMER and doing a bad job). Fortunately, my advisor helped me learn the techniques and Sean approved (he was also happy that HMMER beat BLAST for remote homolog detection).

I also left a postdoc position over my professor's decision to rewrite my paper to juice all the stats- not a specific error, but selectively interpreting the data to make the results look better than state of the art, when they were not.

I would absolutely love to have my "paper correctness AI" mark all those bad reproductions in the literature and outright misrepresentations- it's all too easy to rush to publish and get a lot of attention- especially if your advisor or coauthors are prestigious and mildly unethical.

Reply on news.ycombinator.com