The early days of peer review: five insights from historic reports | Hacker News Reader

The early days of peer review: five insights from historic reports

nature.com

61 pointsrntn32 comments

Intellectuals were writing letters explaining their findings(and asking for feedback) to each other and (math|royal|philosophical) societies all over Europe for 400+ years - shouldn't that count as an earlier form of peer review?

Current peer review is more adversarial.

Also, current peer review is a gateway to tenure, recognition, prestige, funding, influence... When you factor in that intellectuals are also, err, human, you get yourself a nice recipe for disaster.

How many papers are actually replicable? How many are the result of citation cartels or outright fraud?

I remember Shriram Krishnamurthi talking about artifact evaluation in papers [1]. I think that's a great initiative that should be adopted by all fields of science.

1: https://artifact-eval.org/

Even worse, it was originally popularized by Robert Maxwell, yes that Maxwell, to save on the cost of paying an esteemed scientist in the field to sit down and do a serious review.

I never heard about "that" Maxwell (1923-1991) [1].

Paraphrasing Wikipedia [2] [3].

"Robert Maxwell, described as a "British media proprietor, politician, and fraudster," founded Pergamon Press, an Oxford-based publishing house specializing in scientific and medical books and journals. Known for his flamboyant lifestyle, ultimately embezzled hundreds of millions from his own companies.

Pergamon (originally Butterworth-Springer), was established in 1948 to bring "Springer’s know-how and aggressive publishing techniques in science" to Britain. Today, Pergamon Press operates as an imprint of Elsevier."

I had no idea. Beyond wild!

1: Not to be confused with James Clerk Maxwell (1831–1879), creator of "Maxwell's equations".

2: https://en.wikipedia.org/wiki/Robert_Maxwell

3: https://en.wikipedia.org/wiki/Pergamon_Press

This used to be the view among professional historians of science 25 years ago, but the consensus now in the field is that it’s anachronistic: it back-projects present day views of the scientific process onto the past, akin to looking at the first airplanes and calling then “an earlier form of the Boeing 747.” Depending on who you ask, people will say modern peer review arose in the Cold War or at the earliest in the 19th century.

See this encyclopedia entry for a short summary of the issues by a historian: https://ethos.lps.library.cmu.edu/article/id/38/

One of my neighbors conducted studies and published results for Lockheed in Georgia in the 1970's. This was during the development of very large transport aircraft, and solving the associated aerospace challenges. Those studies was usually cross-checked and reviewed by another PHD at another base (Wright-Patterson). The available pool of potential reviewers would have been small, and the profession was insular (US and NATO scientists mostly). Also, no one had a football field size wind tunnel, and computers were primitive/non-existent for simulations. One of them included printed source code. https://apps.dtic.mil/sti/tr/pdf/ADA000431.pdf

Why on earth would someone not call the first airplanes an earlier form of the Boeing 747?

> Why on earth would someone not call the first airplanes an earlier form of the Boeing 747?

Because you need to draw the line somewhere reasonable.

Otherwise, why don't we take your logic one step further and say an aluminum sheet is an earlier form of the Boeing 747 ?

The difference between the Wright Brother's plane and a Boeing 747 is simply too enormous to be a logical "earlier form".

The only thing the two share are the fundamental physics of aerodynamics. And even then, we are only talking the sort of shared aerodynamics you'll learn about at an aerospace museum on a school trip when you get to put a piece of cardboard in a wind tunnel.

Its a bit like saying a donkey is an earlier form of a car. Well, sure, people used to use donkeys extensively for transport. But I think most people would agree calling it an earlier form of a car is pushing it.

The first airplanes are an earlier form of the airplane. The 747 is a later form of the airplane. That doesn't make the earlier airplanes an early form of the 747.

In the same way, cats are a smaller form of a mammal, and giraffes are a larger form of a mammal. That doesn't make cats a smaller form of a giraffe.

I'll bite: what doesn't make cats a smaller form of a giraffe? We just have to squint a little on the spiney level

The definition of giraffe is what makes a cat not a smaller form of giraffe. It has a definition, and the definition isn't just "thing with a spine" or even "mammal".

From dictionary.com: "a tall, long-necked, spotted ruminant, Giraffa camelopardalis, of Africa: the tallest living quadruped animal."

A cat does not fit that definition, no matter how hard you squint.

ITT: Hacker News reinvents taxonomy from first principles.

Perhaps because this wouldn't make sense in contexts where you're discussing what distinguishes the Boeing 747 from other planes. If you're comparing to, say, velocipedes, it makes a lot more sense.

Imagine you are on an archeologic excavation and you find a roman chariot. Would you call this an ancient Honda Civic?

There are some little gems in the article, one emphasizes what a badass Turing was.

As if it wasn’t enough to essentially invent the field of theoretical computer science and save possibly millions of lives in the war, he then decided in 1952 to casually publish in the field of biology.

He had no formal training in biology, and when peer reviewers including the grandson of Charles Darwin commented on his paper “Turing seems to have ignored both sets of comments” lol.

Nevertheless the paper turned out to be groundbreaking and is still considered relevant today. Total badass.

To get primitive...

Science translates observation into abstraction. (That's a big leap. And yes, one could make one's own observation. But nobody has the time for that.)

But is it a good abstraction? (And then we have all these methods for ensuring that.)

Also, why this implicit preference for abstraction? (Yes there's an upside. But consider the downside.)

Everything we experience is abstraction

By abstraction I mean a contrivance of symbols (ie things to which meaning is assigned). Be those symbols ideas, words, patterns of magnetized dots or whatever.

Sight, sound, taste, etc are not that.

(I suppose even words are not necessarily symbols. I mean, if you don't assign meaning, if you don't interpret it that way, then a word is just a unique pattern of squiggles)

When working in research my boss who was very accepted in the field was on many review committees, but I did lots of reviewing (for my failed PhD, "pre-scanning" etc.) - is this the case everywhere? The reviewers by name are not the people doing the actual reviews?

Its pretty common for the top tier research PIs to rest on their laurels a bit and let their sub PIs or post docs (even grad students) handle the work of writing up the grant proposals and even doing things like peer review on their behalf. These PIs basically exist to support the machine running under their name as they can secure way more grant money than if the lab were broken up.

> Doing things […] on their behalf

We train grad students to do reviews by having them do reviews under close supervision, yes. You say it like it’s a bad thing.

Well when I did the reviewing for my PI I rarely got close supervision :) It was more like, 'here's my review', followed by 'I sent it on thanks'

Well it would be one thing if the editor of the journal asked for the grad student to be the reviewer but they asked the PI. Seems a little disingenuous for the PI to pass off that effort as their own. Plus it no doubt takes time away from the grad students other responsibilities.

Most journals that I know require the actual reviewer to be named. Usually it's phrased as "someone who assisted in writing the review." Someone could lie I guess but I'm not sure what they'd get out of that and at that point it would be outright fraud.

Its not named for the public, but the editor specifically looks for certain reviewers who are qualified to comment on the work at hand. They don't just grab random people they look for specific people. And if they are getting a bait and switch out of it instead, well, that's not really honest to the editors efforts trying to get a quality peer review for their journal.

In neuroscience it was common for PIs to review with the help of one or two trainees. Typically not an outsourcing in my experience but I had good mentors and not everyone operated that way.

>> Ferlier says that the introduction of the standardized referee questions significantly reduced the amount of time and effort put in by reviewers. “There’s really this understanding in the nineteenth century and very early twentieth century that the peer review is a real discussion,” she says. “After that, it becomes a way of managing the influx of papers for the journal.”

Screw that. If you want me to put in the effort to read k pages of your conference or journal paper you better be prepared to read k papers of my review.

For some reason Nature and Science never want to discuss the other side of peer review - not that involved publication of a paper, but that involved in accepting or rejecting government or other institutional funding proposals. Let's say the NIH announces $1 billion in funding for cancer-related research - surely the system that distributes these funds is worth a little scrutiny, and it also involves peer review.

Nature and Science regularly discuss these issues. For example, this article from this month:

https://www.nature.com/articles/d41586-024-03106-w

The vast majority of articles on peer review in Nature and Science only refer to journal publication, not funding distribution. Of course they're linked as a publication in Science or Nature helps greatly with the next round of funding applications by all accounts.

It matters outside of academics because NIH funding distributions are linked to exclusive patents for pharmaceutical corporations, which represent a massive public subsidy for the pharma sector. This system also accounts for much of the price inflation in the medical drug markets in the USA, and a lack of research into medical uses of out-of-patent (cheap) drugs. See for example:

https://medicalxpress.com/news/2023-07-nih-spent-950m-basic-...

> "The article in PLOS ONE titled, "NIH funding for patents that contribute to market exclusivity of drugs approved 2010–2019 and the public interest protections of Bayh-Dole," is the first to look broadly at the NIH funding for the basic and applied research related to new drug approvals and the proportion of this research that leads to patents that give manufacturers market exclusivity."

Sure, it's an important area to discuss in public. I took issue with "never want to discuss", which is not accurate and easily refuted by example. You've backed off to "[not] vast majority of articles on peer review", which is accurate.

Reply on news.ycombinator.com