Hacker News Reader

Comment by btrettel | Hacker News Reader

> The one positive of this environment that when you see someone who has succeeded in publishing a substantial amount, it's safe to presume that their work is not entirely without merit. That's not a 100% guarantee, but it's a good heuristic. A post like this implies that the author's personal judgement is more insightful than a process that I know to be enormously thorough and harsh [...]

This has not been my experience. In my own field, I've found that quite a few well-cited peer-reviewed papers have severe flaws that are not difficult to find. I can't call the process "enormously thorough and harsh". I view current academic processes as more of a facade than anything else. Sure, the process probably filters out absolute garbage, but I don't think most people need peer review to recognize the worst quality research.

View on HN

Complaining about flaws in a few papers is like saying “I found a piece of software with a bug” and then claiming software engineering is a broken field based on those examples. The argument about peer review is not that it’s perfect on any given paper, but that someone who has repeatedly survived a publication competition with a ~20% success rate and then got other serious researchers to cite and build their work is probably not a total piker. Everyone makes a few mistakes and papers, but in the aggregate it is a shockingly hard gauntlet to run, whatever your impressions may be from the sidelines.

> Complaining about flaws in a few papers is like saying “I found a piece of software with a bug” and then claiming software engineering is a broken field based on those examples.

My point wasn't a complaint about a few papers, rather, important papers that presumably have had a lot of eyeballs go over them in addition to being peer reviewed. If those papers have serious flaws, then I think that does say something about the standards in academia broadly.

One reason why I don't work in academia is that too many academics resist the idea that there is a quality problem.

Also: I didn't say that my field was broken or anything like that. It's not optimal, but for the most part it's okay. That has little to do with peer review.

> someone who has repeatedly survived a publication competition with a ~20% success rate and then got other serious researchers to cite and build their work is probably not a total piker

Sure, it's evidence that the work is good, but it's not particularly strong evidence in my view.

> whatever your impressions may be from the sidelines.

If you're implying that I haven't published in a peer-reviewed venue myself, you're mistaken. I've never had a problem publishing. I think that's because I agree more with Gauss: "few, but ripe". Unfortunately "publish or perish" reins supreme in academia, which probably explains why so many people have trouble as they seem to try to produce "least publishable units". Borderline papers like that of course will have trouble.

I think tarring the entire field of scientific research by saying "I claim I found serious flaws in a few unnamed papers" is pretty meaningless. Provide specifics!

Based on conversations I've had with tenured academics like yourself, providing specifics won't convince you. You could simply say that you aren't qualified in my field so you can't evaluate what I've said, or deny that I've identified enough problems. Those are reasonable counters, but they won't be convincing to me either.

The most effective thing I can do is encourage you to find problems in published papers yourself.

Another factor in my reluctance is that explaining each case would take a significant amount of time. (Brandolini's law, basically.) I can point you to places where I already explained problems I've found, for example: https://pubpeer.com/publications/95455FA4147A9CBD5EAA5185D21...

That paper has over 300 forward citations over about 25 years. It was published first in a conference, and later in the top specialized journal in my sub-field. I went through a large fraction of the forward citations and could not find anyone who noticed the major error I point out. And it's not difficult to spot if one does basic spot checks.

Let me understand this: you found an error in a paper published in 1998, over 24 years ago. Based on this finding you were able to publish a critique and partial solution in a more recent conference or journal, which seems like an absolutely first-class scientific outcome. (I can only assume that very few people were relying on this stupid model, since the flaw wasn’t noticed for decades.) And yet despite this reasonably good outcome correcting an ancient and apparently little-studied result, you feel that this error indicts the entire field of scientific research… because people have cited the paper?? (Most likely, given the number of citations, without relying on the model and simply listing it in “previous work” sections.)

You are also unhappy that you can’t get a response from the first author [initially I read their Google scholar as indicating they’d left the field, but they haven’t and that was my mistake reading Google Scholar in Korean.] Meanwhile you have not contacted the third (typically senior and advisory) author, who appears to have publications as of 2022?

Yet even if you do the basic legwork here and contact all the authors, I’m not sure quite what result you want from this? I presume you’d like an acknowledgement that an old paper has errors? You’ve already received that through the acceptance of your own paper. Would an admission by the third author help? Do you demand a retraction of that ancient paper and every single paper that cited it since? And how is this the devastating critique of all science that you seem to think it is?

I'm not sure we're having a productive conversation. Perhaps I've been unclear, but I didn't say this was a "devastating critique of all science". I said that I thought my field was "for the most part [...] okay" and I think that's true for science broadly. My main point is that peer review as currently practiced is not as good at quality control as you seem to believe.

My link was just one example. As I indicated, I don't have time to give more examples, and I didn't expect to convince you. I think few academics are actively looking for errors, so unless you are, you may not notice them as often as I have.

> I can only assume that very few people were relying on this stupid model, since the flaw wasn’t noticed for decades.

There are quite a few papers that directly use the criticized model, from the 90s until recently. It does seem that many people blindly accepted a clearly wrong model. With that being said, the other model in the paper is responsible for most of the citations.

> You are also unhappy that you can’t get a response from the first author [...] I’m not sure quite what result you want from this?

The first author is the senior author here. (The conference paper version does not have the other two authors.) I contacted them because if I'm wrong, I wanted them to tell me how so that I could avoid making a fool of myself in my paper. I don't want their paper to be retracted, though a notice that it has severe errors would be helpful to readers.

Reply on news.ycombinator.com