“Peer review” is younger than we think | Hacker News Reader

“Peer review” is younger than we think

sappingattention.blogspot.com

129 pointsPetiver37 comments

The journals published by the Royal Society, where modern enlightenment science was born, was informally peer reviewed by the mid 1700's, and peer review was systematized in the 1800's. The explosive growth of the industrial revolution, the reordering of society in the enlightenment, and everything we call modern and progressive is associated with a sequence of events that is also tightly correlated with the birth of peer review. Most philosophers of science don't consider this a coincidence. Science gave birth to exponential progress, and science itself is dependent on peer review.

Maybe the integration of peer review into practical industries, medical studies, and government regulation came later. But those were cases of taking something that worked astonishingly well and apply it to other domains. It's hardly fair to say that peer review "started" in the 1960's or 70's.

The blog indicates this quite clearly and is speaking of only the formal, modern and monetized usage of it. Science itself is basically useless without some type of peer review in whatever form.

What I'm taking issue with is this:

> But even if peer review is ancient, "peer review" itself is quite new.

Google charts about the history of usage of a particular grouping of terms is not definitive evidence about the history of the underlying activity named by that phrase. Peer review is indeed ancient and fundamental to science and scientific progress. It just may not have been talked about as broadly or referred to under different names before then. It's hard to draw more than linguistic trivia out of that. It's especially bad to make policy suggestions based on that.

But that's the author's point. This current thing we call peer review isn't what's fundamental to science, the broader concept is. The modern for-profit journal system is not the only way to do science, and just because it cloaks itself in the name "peer review" doesn't make it the only sort of peer review that facilitates good science.

There are three activities:

- critical feedback on papers

- hypermonetization (and deleterious siloing) of works often produced with government grants, leading to primarily academics only having ready access to papers, whom don’t see the subscription costs as a burden to them.

- disseminating only papers meeting a threshold to maintain quality of the journal

Maybe scientists and academics should seek to create and use equivalent quality peer / committee adversalism but publish in open forum that operates based on donations.

Academia seems currently over-commercialised: industry getting cheap labor, undergrads going into debt and journals, law libraries and textbook companies making a killing by locking up knowledge behind a paywall. Research papers and case law should be available to the most people as possible for little or no cost.

Why the emphasis on money? The structural organization of science as a hierarchical enterpise with gatekeepers to advancement will result in these problems regardless of funding or commercialization.

Tying funding to it makes them much more acute though. Essentially forcing you to play by these rules in order to live.

Without being forced like this, many deadbeat fake journals would die and places like arxiv would flourish.

That organization itself is a consequence of the funding policy. There's nothing else pushing scientists into a hierarchy, or at least nothing nearly as strong.

Maybe I've missed the author's point, but you've also missed mine. The fact that the phrase "peer review" dates to the 1970's doesn't mean the concept wasn't in use before. It was just called something else ("refereeing" as someone else in this thread pointed out).

Also, the sharp discontinuity in 1970-1980 is suspiciously close to the wide-scale deployment of computers. Probably has a lot more to do with data-availability biases in the corpus used by Google to construct these ngrams data sets from the transition to digital in industry and publishing.

"Peer review" originally meant the scientific concept of reviewing your peers, formally or otherwise. Currently that is not the dominant usage. For example, perelmans proof of the poincare theorem is not "peer reviewed" in the contemporary parlance because it was dumped on arxiv, and not published in a journal with a structured system of presumptive validation. It would be silly to claim that his work is not/will not be reviewed by his peers -- "peer review" in the classic sense.

I'm a bit older than most people here. I mention that because it may be relevant.

I'm entirely unfamiliar with the work you mentioned. So, I'm going to use just your description of events and work with that.

I'd say that their work was currently under review. After people have reviewed it and written about it, I'd say it was peer reviewed. Once it had been heavily reviewed, I'd probably say it had been formally peer reviewed.

Had it gone through a journal, I'd simply say it was reviewed prior to publication. After publication would be where I'd call it peer reviewed, after it has been published and commented on. Pre-publication review isn't very rigorous. I really don't put much emphasis on that.

To me, peer review happens post publication. Formal peer review would be when there is enough interest for people to make a concerted effort to replicate and dig deep into the findings to seek flaws. General Relativity would be formally peer reviewed. The paper you mention would be undergoing peer review.

I am not asserting that my thinking is correct. I am mentioning this because I see a difference and I think your use of 'currently' may be very important in understanding this.

In 1970, I would attend a reputable prep school. I would not complete my doctorate until 1991.

So, I suspect you're onto something with your use of 'currently.' If I understand your comment about the proof of the Poincare Theorem correctly, I'd say that it is (likely) being currently peer reviewed.

To try to sum this word salad up, I don't see prepublication review as peer review in the scientific sense. To my mind, that happens after publication and with varied levels of formality.

So, I suspect you're onto something when you mention current usage. Sorry for the novella and rambling nature of my post.

it's ok. Generally if you post a sketchy scientific statement, say, something by an advocate of homeopathy, a common retort will be, "it's not in a peer-reviewed journal". Of course in the vast majority of cases not being in a "peer-reviewed journal" is a sign that the work is shoddy, but those it comes at the expense of elevating the converse statement - that something in a "peer-reviewed journal" is necessarily of good scientific workmanship.

There is a weak connection between being in a "peer-reviewed journal", having undergone sufficient peer review (in the classical sense), and possibly the irony is that the connection is has a negative correlation with profile at the high end. (it's probably positively correlated at the low end).

Latin and French were the languages of science for quite a while. Arabic before that and Greek before that.

This is ignoring Eastern science intentionally.

They searched for 'peer review.'

I don't see any indication that they searched in other languages, what those phrases might be, and how they accounted for that.

Well, that and it looks like they are conflating two ideas. They appear to be speaking about more modern journals and ignoring that philosophers shared and published their work, sought students, and held their findings up for scrutiny.

So, not just the phrase part that you mention, but also the language used may impact this. I have no idea how to say peer review in either French or Latin and I am a scientist.

Also German, for a short but meaningful period of time at least. Most of modern physics was originally published in German, for example.

Just for the sake of solidity, would you care to have a citation there?

Ironically there's some given in the article.

Isaac Asimov’s New Guide to Science.

The key word is: informally

Formal peer review seems to encourage collective backscratching and discourage new ideas (even though some places do anonymous PR, it's not hard to guess where it's coming from if you're a specialist on the area and familiar with the most prominent labs)

I can confirm that it is very easy to determine who your anonymous reviewers are in specialist areas. In one topic that I used to publish in, I could easily identify one or two of the three reviewers. The main tells are when the reviewer brings up their tangentially related niche interests (e.g. how does this compare to method X or have you considered the effects of <relatively obscure phenomenon>).

Also in some cases (e.g. many conferences) you can know who the list of reviewers are. Just look up the program committee. Only a subset reviews your paper submission, but with the field of possible reviewers constrained to 10-12 people you personally know, it becomes very hard to hide.

One of the key factors missing in this discussion is that there are two overlapping forms of peer review: One which is merely technical, checking for correctness and completeness. The other is a much more subjective form, in which a reviewer is asked to comment on the "merit" or potential "impact" of a given study. The former is often necessary, as honestly there's a lot of crap out there, but more often it comes down to minor details that the expertise of another scientist's specialty can best address. The downside is that this can inspire a certain pedantry and pressure to undertake what is essentially unnecessary formatting.

The latter is essentially the real problem of science in its current state: prestige. There is no reason to not just publish everything in a mega-journal that doesn't consider "merit". It would mean more replication studies and less publication bias against "negative" results (i.e. this drug didn't work as hoped). High-prestige journals also tend to charge correspondingly higher subscription fees, while adding nothing special to the scholarly ecosystem, beyond delaying (often by months) the eventual publication elsewhere. However careers are made on publishing in high-prestige journals.

A recent tweet makes one wonder what the value of high-prestige journals in the first place, if they don't even bother to correct or improve the scholarly record. A PNAS editor's rejection of a paper with greater statistical power and improved methodology, but which fails to replicate a highly cited PNAS paper was on the basis that "in these days of computer literature searches, I see no reason a journal needs to feel compelled to publish follow-up papers on previously published work. The follow-up paper would be found by those interested where ever it is published" https://twitter.com/mendel_random/status/917115714544324609

Another interesting tidbit is that Evidence Based Medicine, what we've come to known as randomized controlled trials with systematic reviews, is /much/ younger than people think. The movement properly started in the 1960s, but only came to fruition after Archie Cochrane (1970s). Still Evidence Based Practice is a contested thing, and very few systematic reviews eventually end up informing policy.

Blind experiments in medicine were carried out a lot earlier. See for example the revolutionary work of Claude Bernard in 1865 [0]

[0] https://en.wikipedia.org/wiki/Claude_Bernard

Sure, but I was specifically pointing to EBM https://en.wikipedia.org/wiki/Evidence-based_medicine

Peer-review is deeply flawed and in its current form merely an advertisement form. We should focus on formatting science in a way that accounts for merit based on reproducibility, falsifiability and paying for real-life usage for recipe invented. Problem is not all science work is technological application.

maybe it's time to augment peer-review with some other stuff instead of completely getting rid of it?

There are many, many BS papers because of the mass-production death-march researchers are told to produce, low p-value and sample size standards and

”Is Most Published Reseach Wrong?” (Veritasium, 2016)

https://youtu.be/42QuXLucH3Q

Which is about an essay from 2005: http://journals.plos.org/plosmedicine/article?id=10.1371%2Fj...

This doesn’t mean adversarial and critical feedback are bad, it means dis/incentives, politics and standards need adjustments.

The problem is not primarily peer-review, the problem is the ridiculous publication pressure we're under. I'm a postdoc in the humanities and some of my colleagues publish 10+ bullshit papers in crappy journals per year - their career is virtually guaranteed, even though everybody knows and agrees that it's impossible to write that much and maintain any level of quality. It doesn't matter how bad or insignificant your work is, if you manage to push out 5+ Scopus-indexed articles per year, nobody will be able to fire you or overlook you in tenure track applications, and I see more and more people gaming the system in that way.

It's leads to a horrible display of mediocricity, of course, and anyone with a genuine interest in substantial scientific research has a hard time competing. Yes, some decision makers still take quality into account, but overall there is a clearcut trend towards chickenshit and bean-counting, at least in Portugal where I currently live.

The term "peer review" has gained usage relatively recently but the term "refereeing" has been in usage much longer. The practice itself, as noted in kobeya's comment, is 100's of years old.

The next revolution needs to be in reproducing what has been "peer reviewed". During my time at a genetics company I learned bad science abounds even in the " top" journals.

One particular example is people "collabing" just to get their name on more papers, which they turn around and use as a marketing tool.

Any time I hear the phrase "I've been published $number of times" these days I tend to think they are worse scientist's not better.

I don’t understand the second part of the title. I don’t see anything in the essay to suggest it can or should go away.

The first part is probably solid: peer review is new. But older scholarship was performed in a very different context - few professional scientists, etc.

But the second part of the title, while provocative, makes no sense to me.

Has the "modern way" of doing science been evaluated and compared against older ways of doing things?

While discoveries do go down in numbers it seems most of past advancements wouldn't have survived modern standards (while a lot of one-off studies do go through today)

This is an important point, but the comparison is difficult. Past advancements occurred in a drastically different time period - few professional scientists, for instance.

I don’t defend current peer review as optimal, but I can’t see an easy way to carry out a fair comparison.

Over what time period?

Modern science starts roghly with the Renaissance and really takes off in the 18th and 19th centuries.

Empiricism owes a great deal to the Bacons, Francis and Roger (no relation), of the 17th and 13th centuries

Typography of this article was not peer-reviewed. They somehow managed to use two different fonts within a single word.

    T<span style="font-family: LMRoman10-Regular-Identity-H;">he earliest usage

While the concept of peer review of intellectual property work products may be relatively new, there is long history of craft guilds applying peer quality controls on physical work products in part to protect the reputation of that class of products within the market.

I feel that this title is putting words in my mouth.

Reply on news.ycombinator.com