Hacker News Reader

Comment by tptacek | Hacker News Reader

Just to clarify: this article is against gold standards, not against peer review. It concludes by observing that peer review is a necessity. The argument it raises is that peer review doesn't mean what lay audiences think it means.

I think they're clearly correct, and that even on HN, in a community that (roughly) prides itself on being scientifically literate, there are broad misunderstandings of what peer review means (during the bogus "Sokal Squared" hoax, for instance, many commenters implied that peer review prior to publication was meant to encompass replication). Also, while I'm not a "scientist", I've gotten to do some peer review work for ACM and Usenix, and even in the little bit of review I did, I seen some shit. There is much less formality and oversight to review than you might expect.

View on HN

You are totally right about peer review being misunderstood as replication. That's why preprints have gained so much traction lately. Things have become so complex that sometimes peer review is only good enough to filter results that are obviously wrong.

But replicating a proof or a complex experiment is way beyond what reviewers can accomplish.

Another problem with peer review, especially in experimental fields, are malicious reviewers. Big PIs typically gain control of a particular subfield by publishing something first, then getting lots of papers to review and blocking said papers and/or stealing good ideas from them.

Big PIs typically gain control of a particular subfield by publishing something first, then getting lots of papers to review and blocking said papers and/or stealing good ideas from them.

That's a bold claim, especially the "typically" part. How can you back it up?

I know it's a bold claim. But PIs that dominate a field employ predatory tactics that are very similar to those used by big corporations. It's just economics.

I have witnessed this in some groups I have been a member of. Very unethical behaviors. As I said above blocking papers in peer review, or even rejecting them, and simultaneously sharing key bits from said papers with postdocs is a routine practice.

Knowing journal editors and reaching publication agreements before papers are even written is also very common, and hardly surprising to someone that has been in the field for some time.

I have absolutely seen this in my field. The top-of-its-field journal editor actually requested our paper for publication and one of the reviewers poured vinegar all over it, creating requirements literally no paper of this kind had ever been held to. We actually sent it to another journal, and got the verbatim same review back. They had sent it to the same top reviewer. Amazingly, said reviewer comes out with a pre-print of a similar paper that addresses some of his own concerns, about 9 months later. Amazing how that happened, hmmm?

The one consolation is that at least we know who's reviewing, who's an asshole, and that this space is as hot as we thought it was.

That sucks. I guess the solution is to publish preprints and move quickly. Then ignore scammy reviewers and publish elsewhere.

Proofs aren't very hard to review, especially because math doesn't have the proprietary problems of wet science so drafts aren't hidden. Such is the benefit of working in a field no one can profit from :-) Math doesn't have a replication crisis; false papers are rare.

Proofs are actually incredibly hard to review. Problems will be found in what had been accepted to be a "good" proof years to decades later. There's a whole movement to move proofs over to something/anything more verifiable (e.g. representing all proofs in Coq--but even then you're relying on the Coq proof assistant to have zero bugs).

Furthermore, the standard for mathematical proof has also changed over time, most significantly in the early 20th century. This led to a number of existing results needing to be re-proved (or thrown out! Some were incorrect!).

Exactly what qualifies as a proof is a FASCINATING debate. Mathematics is created by consensus, just like all other knowledge.

What are some examples of proofs that were originally accepted and then later shown to be incorrect?

Here are a few of them:

https://mathoverflow.net/questions/35468/widely-accepted-mat...

Proving a math problem is mentally challenging, but there are other interesting definitions of hard. In medicine, for instance. None of it requires particularly fancy logic. But can you cite any instances in math here you had to invest, say, $10-15M and collect data from every available human patient at multiple hospitals over 2-3 years in order to replicate?

Is there any proof peer review has increased scientific productivity? It is relatively new, and I don't see any reason prima facie why it would work better than an editorial teaming willing to delegate when necessary. Anecdotally, I see lots of academics complain about very important results being unable to make it through peer review. Ralph Merkel's early groundbreaking cryptography work was suppressed for years by unthinking peer review. From Merklel:

>My first paper (and, in fact, the first paper) on public key cryptography was submitted in 1974 and initially rejected by an unknown "cryptography expert" because it was "...not in the main stream of present cryptography thinking...."

Isn’t peer review at most journals done by “an editorial team delegating as necessary”?

I married into the field, didn’t work in it, but that’s what I’ve observed from the side.

The ACM and Usenix peer review work I did involved rotating teams of volunteers and very little guidance from the publication/venue.

Well yes, but the review power (both nominally and in practice) does lie with the publication/venue editor[ial team], it's just that they have decided to delegate almost all that power to rotating teams of volunteers, because they can, and because they want to for obvious practical and financial reasons.

A system with an "editorial teaming willing to delegate when necessary" simply becomes the current system as the editorial team is willing and able to delegate, and it's neccessary pretty much all the time because of resource constraints - because being an editor generally also is an unpaid volunteer position done in addition to your normal work in academia.

  Is there any proof peer review has
  increased scientific productivity?
  It is relatively new

According to [1] "The council minutes of 1 March 1665 made provisions for the tract to be revised by members of the council of the Royal Society, providing the framework for peer review to eventually develop, becoming fully systematic as a process by the 1830s."

Peer review is a relatively new procedure in the same way that California is a relatively new state :-p

[1] https://en.wikipedia.org/wiki/Philosophical_Transactions_of_...

However, some famous papers like Watson & Crick's DNA model were published without peer review.

You can publish today in PNAS without review if you are a member of the national academy of sciences.

Not exactly any more. It's true that it used to be that a member could just contribute their own manuscripts which would be published as is without review. But after some unfortunate cases of questionable stuff getting published, even the contributed track has to be peer reviewed these days. However members still have leverage in that they can choose the reviewers, which obviously could be abused.

The concept is certainly old, but it hasn't been a systematic part of the "universal scientific process" for more than a few decades.

Most revolutionary scientific discoveries were made without it.

"Peer review" nowadays refers almost exclusively to prepublication review by people other than the editor. This was still relatively rare at the beginning of the 20th century.

If you name a famous publication prior to 1900, it almost certainly was published without peer review.

You are thinking of journal review, which hijacked peer review until arxiv and the internet in general rescued it.

Yup exactly correct. I've also been a reviewer from time to time on various ACM and IEEE papers and there are clearly authors who are trying to get reviewed by their 'friends' rather than by objective reviewers. And there is also reviewer biases, I reviewed a paper where one of the other reviewers was extremely harsh, and that reviewer turned out to have a stake in the outcome. Now a good journal is supposed to curate these sorts of conflicts of interest but my experience is that isn't always the case.

To be cynical, sorting through conflicts of interests would take more effort and Elsevier wouldn’t find that profitable. More realistically running an unbiased journal will always be difficult unless human nature changes significantly. Nevertheless it’s still worthwhile to do.

Same as you, I’ve had reviewers reject research based on a reviewer’s stake in the outcome, and also had good reviews with constructive feedback. As a reviewer I’ve also helped "tear apart" an article that had good results but was poorly structured to help them clean up their article.

Getting a harsh review stings but the (normally) impartial feedback can be invaluable. I've come to believe that only by learning how to pass your articles by peer review do you learn how to write a good article.

Even in medical research, legacy publishers do precious little to respect even the very modest standards of the field for the disclosure of conflict of interest. https://bmjopen.bmj.com/content/bmjopen/9/7/e029796.full.pdf

What exactly was bogus about sokal squared? They made stuff up, they got it peer reviewed and they got it published. Yeah the journals weren't great but none of them were publication mills and not everything published gets published in Nature.

I wouldn't consider nature to be necessarily so reputable either. If you want an almost unassailable example of peer review, check this out:

https://en.m.wikipedia.org/wiki/Organic_Syntheses

Interesting. I'm not in academia so I guess that's showing here.

In my field we say "it was published in Nature, but it might still be correct".

The underlying problem with Nature is that they pick the flashy stuff AND present it in a very very condensed matter -- removing all the important technical details -- to make it look exciting for a wider audience. For the few hundred people actually in the subfield the best course of action is often to ignore the PR piece in nature and go hunt for the technical paper with all the details that is published in parallel in a "lesser" journal. That is the one the tells what the actual progress was and what you want to consider for your own work.

There's also Andrew Kelman's renaming of PPNAS, the Prestigious Proceedings of the National Academy of Science, where Prestige is used instead of Truth.

> The underlying problem with Nature is that they pick the flashy stuff AND present it in a very very condensed

I don't object to the condensed part so much. When I was a physicist, people had a similarly dim view of Nature. But Physical Review Letters also published four-page papers. And PRL was definitely a bigger prize than the long form PR(xyz) papers.

Now if you happened to want to drill in and do work based on someone else's, then you would no doubt prefer the long-form. But if you are keeping up with the field, then the 4-pager is much preferred.

You should be able to lay down what the actual science is in a small space. And if it's a theory paper, that space is enough to have a few equations that the reader could then puzzle out justifications for herself.

"it was published in Nature, but it might still be correct"

We said that in biochemistry as well. Also science. There are some huge stinkers there. When arsenic life came out we were like, of course, it's horsedung, it's in science!!

Indeed. Nature ranks very high in the retraction index. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3690355/figure/...

There were a couple issues with the Sokal Squared hoax. The biggest one was that they relied a lot on their own definition of "obviously absurd" rather than using some form of empirical or unbiased "made stuff up" generator.

In the process, they wrote some papers that, especially if you weren't assuming the person made up the data underlying them, were...middling compelling.

I read one of the papers - the dog park one - and my conclusion was much the same as one of the peer reviewers. That it was an interesting concept, presented slightly too stridently (a common thing with papers written by say...students), and interesting enough as a frame to jump off from.

I must compliment the creativity of defending someone from having published nonsense by claiming that it wasn't sufficiently-scientifically-generated nonsense.

If someone not in the field can write a "middling compelling" paper in a distant field while trying to be absurd, then that still means something. If some jerk (basically) can just scribble a "publishable paper" off in some semi-major discipline while literally doing the opposite of trying, then clearly that semi-major discipline has very little value to it. We're not creating these large infrastructures with journals and university libraries and peer review and the Imprimatur of Scientific Credibility and credentialed university graduate programs and millions of dollars in grants for things any idiot can scribble down. We've got the Internet at large for that already.

If it's by design or somehow correct that peer review can't catch these things, then my opinion of the situation is even worse than if they were merely a demonstration of things slipping through a net, and peer review on any level is apparently utterly worthless.

Your defense may be superficially interesting, but if taken seriously, only deepens the problem.

They weren’t outside the field though. They did a deep dive, learned the language, and relied on a definition of “absurd” rooted in their own ideology rather than reason.

In doing so, they ended up mimicking the field too well.

Again... even that is still pretty serious condemnation. If you can "deep dive" that many fields to the point that you're fooling the gatekeepers, a "deep dive" must not actually be all that "deep".

I mean, we can try to puff up Sokal as some kind of massive genius who got to publication-quality skill levels in multiple fields in less than a year, but... in that case, why shouldn't I accept his take on the results, since he's apparently that much of a genius?

There just isn't a way around it. If it's that easy to get to "publishable", it isn't worth being an academic discipline.

Compare someone in one of those fields trying to publish a particle physics paper by just imitating what they think are the important characteristics.

Note we’re not talking about Sokal, but a separate hoax that decided to call itself that and which was distinctly less impressive. They started, at best, one field adjacent from what they targeted.

I’m not at all surprised they managed, after devoted effort to do so, in having middling success getting papers into some okay journals.

Couldn't agree with you more. I too have done peer review, and the most disappointing thing in the world is when you point out something that is inaccurate and the response from the publisher is, "if we made it accurate there is too much nuance for the layperson to understand, so we'll keep it the way it is even though it's wrong because they'll get the gist of it".

There is a lot of frustrating things in being the reviewer, but "that would be too much nuance for a layperson" is not one I have encountered at any of the four journals I have reviewed for.

Technical books tend to get peer reviewed.

Wait, when does this happen. In my experience, the sort of document undergoing peer review is rarely the sort to be read by laypeople to begin with.

There's sometimes peer review on popular science books. Maybe he reviewed a few chapters of "Netflix: What happens when you press play?" (https://www.amazon.com/Netflix-What-Happens-When-Press-ebook...) or something.

Ha that's funny. I literally gave a talk with that title. But no, I didn't review that book. But yes, it was similar popular science type stuff.

Technical books tend to get peer reviewed.

As a machine learning / probabilistic programming friend put it, (journal) peer review is boosting with three weak learners.

Better than nothing, but very far from ironclad. Only replication really verifies scientific findings. Everything else is just window dressing.

(I say this as a regular reviewer; for whatever reason, this particular week I’m reviewing for both Science and Nature.)

Reply on news.ycombinator.com