Peer review is an honor-based system (2008)
lemire.me
lemire.me
The peer review is there to help the journal maintain its reputation by preventing the publication of sub-standard stuff. Period. Sub-standard can mean uninteresting, incomplete, poorly written, or whatever the journal is aiming for. It is not there to safe guard the integrity of the literature against erroneous results...it's purely self-interest on the part of the journal.
In reality, a rejected paper will just be submitted elsewhere until it is eventually accepted. The authors cannot afford to spend 1-2 years worth of work on a project then have nothing to show for it, just because a reviewer didn't "get it". So authors will keep submitting it (hopefully with some improvements based on past reviewer comments, but maybe not) until it "gets through" somewhere, and eventually nearly everything gets the seal of "peer review".
> There is SO much more I could write on this subject, but I'm trying to stay on point (-:
It seems strange to me that we built institutions on the idea of filtering in/out applicants based on relatively arbitrary criteria, and then express shock/surprise when the reward systems inside that institution are.. basically the same?
There are parallels everywhere, e.g. scientists feeling they must get positive 'groundbreaking discovery!' news reporting about their publications, not just actually doing impactful work, in the same way good grades aren't enough and you need some other impactful story to tell in order to be accepted to many schools.
All of it can be traced back to money, money, money.
I think it is impossible to use anything other than noisy signals in many of these processes. I'm not bothered by the noise. But what I am bothered by is pretending the noise doesn't exist or trying to convince people it is a feature and not a bug. Why not just admit the process is noisy and that we're just doing our best? These processes may still be frustrating, but they will be less so and it leaves the door open to fixing issues if they are actually fixable.
With the way we publish science, I find it very odd. We are doing essentially the same thing we have done for centuries with only a few minor tweaks. But unlike the past we don't have many of the same constraints; such as, printing, lack of color, distribution/storage, communication (which is arguably the main purpose of papers), video, and so on. Why are we stuck in the 18th century with a sprinkling of 21st century (plots/graphics and arxiv)? We can track data, reproduction efforts, failures, challenges, and all that stuff. Is not the point of science the pursuit of knowledge? If so, it seems like we're being pretty inefficient. I think there's just something funny about how we publish papers on things like computer graphics and video but constrain our communication methods to a piece of paper.
Peer review is not inherently terrible. Exploitative rent seeking publishers that commoditize academic careers and outputs, and hold knowledge to ransom, are the problem.
Please, everyone, stop publishing in journals that do harm. Think about what the impact of where you publish a paper is and align your choices with your values.
It's not no bar. Peer review adds some credibility to a paper. And the venue does as well. It's just less credibility than the popular imagination assumes.
A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes.
As a data point in ML conferences, I had a reviewer give me a strong reject with only a few sentences. First was a generic "not novel" with no further explanation. The rest was a complaint that my paper had serious flaws and pointed to an intentionally redacted link and a broken cross-reference. We reported them to the AC and the other two reviewers were borderline and weak accept, but both had low confidence (of course the inane reviewer was high confidence). The result was that the reviewer's rebuttal was much more angry and the weak accept said "authors addressed my concerns, but I've decided to lower my score."
I tell this story because at a certain point it is not a problem with bad reviewers, it is a problem with bad management. The most important thing that a system can have is self correcting behavior. A system that gives a bad faith actor a slap on the wrist and allows them to then escalate their bad behavior and influence others is not a well functioning system. Granted, this is just one example, but I've seen many others (such as a theory paper being rejected for lack of experiments). I've started collecting author tweets I find about these kinds of situations, but I think there's a lot more that aren't easily conveyed on social media.
I do really want to promote discussion of these things because I think we should always be improving our systems. You're right that nothing is every perfect, but we can still recognize flaws without being consumed with the impossible goal of perfectionism.
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2399580
was shitcanned by a Microsoft reviewer (if you're the one and you're reading this: GFY)
First of all, he ignored it for six months, and it wasn't until I begged the editor that he got it moving. Then he gave one round of critique, and when I addressed it, he said, "the patent law changed" and closed the process. He knew nothing about patents.
As I said, the editors should not defer to these douchebags. Fortunately I don't need pubs for my career, but some people do.
I will never understand why in CS we use a conference system as our indicator of merit. It is an antagonistic zero sum game. There is no recourse for giving a bad faith review and you are incentivized to reject works. It's easier to reject and thus less time consuming -- every paper has flaws and limitations, just point at them and ignore context--, you marginally increase the chance of your own paper getting in, and venues use acceptance rates as the main measure to indicate their level of prestige (if I cared enough I'd just get LLMs to spam papers to abuse this). Every part of the system that does something good and beneficial to the scientific community relies entirely on people acting in good faith and against other incentives they have, favoring the purity of science. That clearly doesn't scale and clearly sets a stage for bad actors to overwhelm. I think we only continue this because they've successfully convinced good faith actors that there is no other way and that it isn't as bad as it is. I just wonder how much money and time is lost due to all of this (before we even account for the ridiculousness of charging for what arxiv does for free. The paper is given to them for free as well as the reviewing is done for free, and much of the organizing committee and all that is also free labor. Something this important shouldn't have so many similarities to a scam that seeks to extract money from the government and universities).
In my experience, journal reviewing is much worse, and especially much more corrupt, than conference reviewing.
At conferences you get bad actors with almost no oversight or accountability, true. But at least they typically don't have an axe to grind, because accept/reject decisions tend to be binary (there is no "revise and resubmit", at most optional recommendations for the final version that aren't effectively enforced).
At journals, the "revise and resubmit" option gives reviewers and editors a lot of leverage to push papers their way, and I have very often seen reviewers tacitly and subtly hint that the paper would easily be accepted if the authors included three or four references to papers by Author X (which are irrelevant to the subject matter, but of course, Author X will be the reviewer or a friend). Sometimes it's clear that the reviewers didn't read the paper, their sole interest on it is that you include their citations, that's the reason why they accepted to review and all they look at. Editors could in theory prevent this, but often they just echo reviewer comments, and sometimes they are complicit or even the originators of the corruption (it's also typical that the editor asks to cite a bunch of papers from the same journal, even if they're not related, because that inflates impact factor). In any of these cases, since the author is in a position of weakness and there's always the career of some PhD student or other down the line, one doesn't typically dare to complain unless the case is really blatant (which is typically not, because they know how to drive the point subtly). It's easier to go along with it, add the citations (knowing that they will lead to acceptance) and move on.
This has happened to me very often and I'm not talking about shady special issues in journals that print papers like sausages, I'm talking about the typical well-regarded journals from major traditional publishers. In conferences I've gotten all sorts of unfair rejections, of course, but at least the papers I've published accurately reflect my views, I can stand behind them. In journals, maybe half of my papers have this material that doesn't make sense and was added to appease a corrupt reviewer.
I find that many CS authors who haven't had to publish in journals have a "grass is always greener" mentality, and expect that if we moved to journals we would find a fairer review process... and if at some point we do so, they will receive a blow of reality (not saying it's your case, of course. There can also be people who have published in journals and disagree with me due to different experiences! And there are some journals that don't tend to engage in that kind of corruption, only that there are not many of them).
It's a really weird political game and I just think we need to move to a system where we either have reviewing seen as a collaborative/allied effort (as opposed to adversarial) or we rethink the whole system entirely and let reviews happen naturally (i.e. submit to OR and allow comments). It's just very clear that a system can't scale if it relies on all members acting in good faith. Especially when there's high amounts of competition.
Edit: one thing I find interesting is I contextualize different than other reviewers. Like I see a experiment section where they have a node of 2080Tis and think "okay, students or small lab. Compute bound, so do the experiments they performed optimize under those conditions?" Whereas I think most reviewers don't contextualize in this manner and so I see many act as if every lab has access to many A100 nodes. I think this matters because the small labs can still do essential work, but we just need to recognize the signal may not be as strong. Their workload is probably higher due to lack of compute too (and I subsequently tend to see deeper analysis and less reliance on letting the numbers do all the talking). I don't think the GPU poor "aren't contributing" because they can't, I think they can't because gatekeeping. It's insane to expect academics to compete with big tech in a one-to-one comparison. If all you care about is benchmarks then compute always wins.
Was this back in 2014, or more recently?
The thing is, "the patent law" had not changed with regard to obviousness (103). CLS Bank v. Alice which I think you're alluding to was about 101.
I'd have to go ask my dad about how all this works specifically lol
103 is about obviousness. 101 is about patentable subject matter. I actually had an application rejected on 101 grounds after CLS Bank.
For the filler papers nobody cares after they get published, they might cite them for the sake of completeness.
The seminal papers though will be replicated and any hint of BS will be revealed.
Firstly, this reminds me of a blog post about what it means to get a PhD [1]. The problem with that view is that the circle of human knowledge is drawn as a solid, while in reality it is very holey. So seminal papers push the border outward, while filler papers fill in the holes. Both valid and necessary, but general public only cares about advancing the borders.
Also, filler papers are needed not only for the sake of completeness, but scientists need papers to demonstrate productivity. E.g. graduate students needs papers on their CV so they can prove to their next employer that they are productive and capable of serious research. So these filler papers also "fill" the role of "filling" up people's resume. The public seem to think that professors publish papers only to announce to the world about their great discoveries, but 90% of the time its filler papers just reporting on business as usual type progress.
[1] https://bretcontreras.com/the-illustrated-guide-to-a-phd/
1. The hoax authors didn't submit gibberish or anything, like with Sokal - they submitted coherent papers that they considered absurd, but which they expected the journals to find reasonable. As such, they only thing they can even pretend to have proven is that they and the journals disagree about what's reasonable.
2. IIRC, every one of the initial hoax papers was flat-out rejected! If the authors had any backbone they could have stopped right there, and published a paper called "These grievance journals aren't as bad as we expected." :D But they didn't - they set out to iterate on each paper until a journal accepted it. And if your experimental methodology is "keep trying different things until X happens", you can hardly pretend to have proved something when X happens.
I mean, I agree with the premise the hoax authors were trying to prove. But their "experiment" was silly, and I think it's dishonest of them to pretend it proved something. Meh. Meh!
Peter Thiel in this recent talk argues by inference that this problem is likely deeper and more pervasive in hard sciences than in the humanities. https://youtu.be/lO0KH-TgvbM?t=22m20s
The time Thiel pines for is simply the time when the US government spent more money funding science, so that there was less competition to get funding. https://ssti.org/blog/changing-nature-us-basic-research-tren...
Actually if you dig up the authors' initial article about it, IIRC at the end there's a line like "so what does any of this actually prove? Well, we'll leave that for you to decide." If the authors can't say what their experiment proved, I think it's reasonable to conclude it didn't prove anything :D
Isn't this a pretty good example of how crude the hoax was? Of course some things that Hitler said won't be awful if you literally change the key words in the sentences. Imagine if I played this trick the other way round. I take something innocuous that a politician has said and replace random words or phrases with 'National Socialism' and 'Jews'. Suddenly what they're saying seems really controversial! And this shows...what?
I’ve never heard anyone describe Mein Kampf as coherent. The ramblings of an asshole clearly abusing stimulants would be closer to the usual.
>Of course some things that Hitler said won't be awful if you literally change the key words in the sentence
The idea that swapping around the group being discussed is all it would take to make it not awful is, well, novel if nothing else.
Edit: What the hoaxers actually say suggests that the passages were altered quite substantially: "The last two thirds of this paper is based upon a rewriting of roughly 3600 words of Chapter 12 of Volume 1 of Mein Kampf, by Adolf Hitler, though it diverges significantly from the original.". And you can read the paper here: https://newdiscourses.com/feminist-mein-kampf/#solidarity
From the link you provided - “Accepted by Affilia , August 21, 2018”
Don’t lie.
We're not in youtube comments here - assume good faith and be civil.
I did.
Twice in two comments foldr quoted from a source. Both of those quotes are literally right next to another sentence in the source which is exactly the opposite of what foldr asserted.
The Affila wikipedia page section titled “Grievance studies affair” is four sentences long. The sentence right before foldrs quote, which is the first in the section is this.
>In October 2018, it was revealed that the journal had accepted for publication a hoax article entitled "Our Struggle Is My Struggle: Solidarity Feminism as an Intersectional Reply to Neoliberal and Choice Feminism."
The New Discourses link does say the paper was rejected from a specific journal in the first footnote which is . The second footnote says the article was accepted. Both footnotes quoted here:
> Peer reviewed and rejected by Feminist Theory
> Accepted by Affilia , August 21, 2018
At this point I am no longer sorry for not assuming good faith from someone arguing that a reformulation of kompf could be “not that bad”.
> We're not in youtube comments here
Be civil.
>harping
I don’t accept the characterization of being thorough as harping, or that anyone would need to be unrelaxed for that. Is that ironic with me being told to assume good faith?
Not saying you're doing it but these rules of thumb don't scale. This is totally a valid tactic in internet fuckary.
Making shit up and attributing it to a source anybody could but nobody will verify is high-school Essaywriting 101.
The point of 'assume good faith' is that it's very easy to convince yourself that semi-anonymous people on the internet are liars. Most of the time they're not. It's very easy to inadvertently make factual errors. Published books that have been carefully reviewed are full of them. What chance do random morons ranting on the internet stand?
Sorry, but assuming good faith is like supporting free speech - if you only do it for arguments you agree with, you're not doing it at all. ;)
I am not sure this is obvious or even true. Can you provide one example? "Gas all the <X>"...what? capitalists? murderers? rapists? pedophiles? It doesn't even make sense for them, what else is there?
inb4, no you don't want to gas these people, you want to fix them if possible or jail them otherwise.
“We must fight against the influence of special interests in American politics.”
“We must fight against the influence of Jews in American politics.”
You may or may not agree with the first statement, but it’s hardly in the same category as the second (which is deplorable). It seems to me trivially easy to construct many more such examples.
The hoax paper replaced references to Jews with references to privilege. It’s hardly surprising that rants against an abstract concept are less offensive than anti-Semitic rants.
> “We must fight against the influence of Jews in American politics.”
To be fair, some people do use the former language to mean the special interests of "Jews." No all, but dog whistling does exist and dog whistling is explicitly about covert language. I don't think the special interest groups reference is a great example because it is a common phrase that is used to mean a lot of different things and isn't uncommon in various groups using covert language.
I do see what you mean though. There's something inherently gross about juxtaposing two sentences like that. I'm not doing it to suggest any equivalence between them. The point is exactly the opposite (and I would have thought an uncontroversial one): that switching out the major vocabulary items in a sentence can take it from being innocuous to offensive or vice versa.
Actually I think that's explicitly the issue and a main part of my point. In fact, dog whistling or other type of coded language typically depend on ambiguous language. It's literally because language works like an autoencoder. There's what you have in your head that's encoded into what you say and then decoded. The coded language comes through a learned/tuned decoding.
My point is that such phrasing isn't inherently innocuous. Vague language is always inherently dubious. The juxtaposition just makes it more obvious in this case, but it's always true. It may not always be dog whistling but in the least dubious case it lets people fill in whatever they want. That's why I'm saying it isn't a good example.
I'm not sure why you would be so caught up on "Jews" specifically when there are plenty of other targets that can be used. But yes, that term has been used to refer to Jews the same way one might say "Hollywood types" or "Bankers." I can't think of a specific example, but I think that's a bit high of a bar as I don't have an eidetic memory and it's not like I've been keeping a log of every time some mentions special interest group. Like most people I only remember the broad concepts and general notions. If you're aware of a way to easily Google or search this information, I'd be quite interested to know, because that's a very useful task. In the mean time, I'm sure you could find some if you dug around more around the Ukraine war, with both US right wing and Russian news platforms being the most likely place to find these. There's a lot of anti-semetic conspiracies revolving around Ukraine (with it even being the Russian's main initial propaganda) so that's why I suggest looking there.
By the way I'm not 'caught up' on 'Jews'. It's just that 'Jews' is one of the words that the authors of the hoax article replaced.
From what I have read so far, they showed that if you completely fabricate the data and follow the advice of peer reviewers, you can end up at a published paper.
Thank you, Captain Obvious.
Had these valiant warriors heard of Diederik Stapel, they would have known that fraudsters already made this point. Extensively.
These fighters for academic rigour seem to have lost sight of their own academic rigour in this "experiment".
Peer review is insufficient to catch fraudulent submissions. This is not a surprise as much as a misunderstanding of what peer review is.
To be clear: to show something, the authors should have shown that the ideas were obviously false, not that fraudulent data that supports findings would go undetected. Einstein's famous works were considered patently absurd at the time. The only way to substantiate or refute such ideas is with data. An experiment that abuses this to "show" anything is, itself, patently absurd.
The problem with publishing elsewhere is that many people unable to objectively evaluate the validity of the theory juge its value by the reputation of the journal. Also the other journal will most probably have less visibility. There is thus a higher chance that the other theory will not be reported in reviews.
There are many assumptions in the above comment that I can't agree with. But my experience is with physics, not computer science.
doing this actually requires a costly investment in deep understanding of the research and the literature around it, to know if it is sound and high-impact. but nobody wants to make this investment, and they've mostly convinced themselves that just free-riding off peer review is good enough.
of course it is not, flawed papers make it through peer review all the time. also this outside use of peer review introduces extremely strong distorting incentives -- an even hugely greater desire to be published in specific prestigious outlets -- in the face of which peer review is not really adequate to catch misconduct.
I think the ideal form of peer-reviewed journal is unfortunately logically impossible: its contents would be completely open access, costing nothing for scientists who want to build on the results. But it would require an extremely expensive subscription to find out who has published in it, the money going to fund extremely thorough peer review, so that outside decision-makers can't try to free ride and distort the incentives.
A paper is a brick. A policy is a building. It is not a problem in the brick manufacturing field, if people keep trying to build houses without any mortar.
(Just to be explicit, I think you are right on the money and just wanted to elaborate/rant).
That's not what peer review means. If that were what it meant, it would have no purpose - the people you describe don't benefit from the paper they're reading having been through peer review. They already understand the domain.
Compare https://genomesunzipped.org/2012/08/the-first-steps-towards-... :
> The solution to this problem relies on a simple observation – in my field, I am completely indifferent to whether a paper has been “peer-reviewed” for the basic reason that I consider myself a “peer”. I do not think it extremely hubristic to say that I am reasonably capable of evaluating whether a paper in my field is worth reading, and then if so, of judging its merits. The opinions of other people in the field are of course important, but in no way does the fact that two or three nameless people thought a paper worth publishing influence my opinion of it.
The only goal that peer review serves is letting people who aren't peers and don't understand what the paper is saying believe that they can tell whether the paper is good or not. They still can't, but because of the peer review system, they feel that they can.
I don't agree with this. There is deception in its name: peer review. Even "preprints" (things on arxiv and not accepted in journals or conferences) are still read and evaluated by peers. LK-99 is a great example of this, and that's not an uncommon process. The problem is that venues perpetuate the idea that being published within them is a mark of validity. But, like you suggest, it isn't. And "peer review" doesn't mean that it is just okay for peers to read it because we're reading each others works long before they are released in venues. Unfortunately most people (including researchers) don't have advanced bullshit detectors or nuance parsers. These are often fairly narrow and there's not that strong of alignment between the subject matter that a reviewer is reviewing and their niche domain expertise. I agree journalism and politicians need to do better, but so do we in the scientific community. There's more nuance. It is definitely a problem with the brick manufacturer if the marketing team implies you don't need mortar or even says "generally you don't need mortar."
I don't think it's limited to entities outside a scientific community, because many of the people who want to use published research to set public policy are themselves scientists (often the same scientists who published the research).
OK, but those are the only use cases for peer review.
> Every time some one comes up with grants for reproducing published stuff the silly idea finds enthusiastic support.
suggests that you think replication isn't useful. Or, am I misunderstanding what "the silly idea" refers to?
Therefore the recent super conductivity papers may well end up being very important, if only to stir up the community to action.
Einstein for example had public resistance to his theory of relativity in the beginning.
More recently, it took years for the medical community to get over its opposition to the evidence that common peptic ulcers are caused by bacteria (which earned two Aussie physicians the 2005 Nobel Prize in Medicine). [0] A NY Times correspondent (and physician) wrote in 2002, "I’ve never seen the medical community more defensive or more critical of a story."
(Much of the opposition was interest-based, e.g., surgeons who didn't want to lose lucrative fees for operating on ulcer patients who could instead be treated with inexpensive antibiotics without the pain and recovery time of major surgery.)
[0] https://www.nobelprize.org/prizes/medicine/2005/7693-the-nob...
[1] https://en.wikipedia.org/wiki/Timeline_of_peptic_ulcer_disea...
Peer review often fails to catch cheaters, too, as the infamous Jan Hendrik Schön case demonstrated (from 2000-2001 the Bell Labs researcher published 9 papers in Science and 7 in Nature, all of which made it through peer review, all of which were later retracted on grounds of fraudulent data manipulation). In the long run, the scandal did improve the field, as all publications on microelectronic graphite etc. devices now require electron microscopy proof that the claimed devices actually exist. Note current AI technology allows for data fraud and image manipulation that's much harder to detect than in the past, though.
In some circles, the two first pieces of feedback on a paper draft, from a nominal co-author:
1. Cite [my people's various loosely related work].
2. Cite [particular researcher in this niche], who'll probably be a reviewer.
Don’t use them as the basis for rewards.
Don’t use them as the basis for policy
If they are nothing more than the the starting conversation of scientists then we should not put much value in them.
Let’s not play the have it both ways game where scientific papers are given deference and then when there is cheating just say li g that scientific papers are just a conversation starter.
What about the use of papers as metrics for advancing in academic careers?
Career advancement could be based on market forces (e.g., productive researcher on something valued, if they're not given a sweet enough deal by one university, another university can snap them up). But how do buyers in the market get the information about how valuable a given researcher is? (Word of mouth? CV summary without pubs list? Job talks? Interviews? Twitter?)
This sounds more like textbooks.
Journals are usually intended to be venues for experts to talk to each other, not even "professionals in the field".
There are some journals (sometimes called "translational") that try to bring scientific results into practice, though these are often limited to fields where "practice" is its own huge body of knowledge that doesn't always overlap with "research" in that field (e.g., Medicine).
This made sense when communication was slow and the back and forth of discourse could take weeks to months. But publishing your work in a way that thousands of other readers can consume it is easier now that it's been ever, so it stands to reason that sharing the results of scientific works may yield better outcomes if it takes on a different form.
It's a potentially important claim
Peer review is just a sanity check. Authors and scientists and studies and papers are important or not independent of the peer review process.
The question is: Who benefits the most from preserving the dying "peer review" system? Who loses the most from the transition to open debate? Short answer: Publishing houses and conference organizations. Neither wants the status quo to change.
[a] https://michaelnielsen.org/blog/three-myths-about-scientific...
For everyone else, there is peer review. Attention economy is unforgiving and benefits the elite. With peer review, everyone's work gets at least a minimum level of attention. Which may then lead to more attention if the work is worth it.
Every serious proposal for replacing the current peer review practices must have the same feature. When someone submits new work, there must be people who have to review it. Fully voluntary reviews don't work.
One of the biggest advantage of peer review is it allows obvious corrections from a 3rd party to occur without wasting everyone in the fields time. Gatekeeping isn’t the point, increasing the signal to noise ratio is.
Some places require an anonymity period, so you have to wait until that ends to submit your paper to arXiv.
The peers that will review your work likely know about the paper you submitted already, because they work on related work themselves and sat through your conference presentations. Most of them want you to publish your work and will provide a good/non-adversarial review of a paper.
Sometimes though, your paper hits too close to home for them, then they will try not to get it published or will slow walk the review so that their own work can come out before yours or at the same time.
On top of that, you have journal editors who can see everything about the process and can decide to ignore a good/bad review to fit their ideas about the paper itself and to fit the overall vision they have for the journal for the coming publication schedule.
How often does this actually happen? Can't say I've heard of people doing this.
We could also take steps towards helping establish high reputation for certain papers by introducing a mechanism other than citation count. Maybe some kind of stamp of approval.
A problematic aspect of non-public peer reviews is waste and political battles. I have witnessed first-hand how big names in a particular field reject articles from incumbents that are perfectly sound just to delay their publication and/or to copy them.
A public review introduces some skin in the game and avoids this kind of behavior, as well as rejections or requests to make changes because of reviewer incompetence. It also avoids the opposite thing, blind acceptance of flawed studies.
Also, all 19 journals published by EGU (European Geophysical Union)
Becoming a reviewer should still be invite-only and the system should keep track of the reviewer's identity behind the scenes to monitor for abuse, of course. The review can include coarse-grained reputation signals like "has reviewed 100+ papers in the last 5 years".
It might be worth embargoing reviews with a fixed time delay before making them public to prevent bandwagon effects and disincentivise review plagiarism tho. The reviewer's identity should be deanonymised too after something along the same time scale as when the paper author's identity is revealed.
"Is there hard evidence that the grant peer review system performs significantly better than random?"
https://academia.stackexchange.com/a/128343
In short, 'freedom of research direction', aka 'blue skies research'[1] is steadily becoming a thing of the past, as grant managers and politicians and academic administration teams increasingly take the view that they're the ones who should be directing what kinds of research are done, rather than the academic researchers themselves. This is enforced by a grant system that narrowly defines how the funds can be spent, meaning that your average academic researcher has been transformed into a corporate drone following orders from the executive floor.
[1] https://en.wikipedia.org/wiki/Blue_skies_research
This transformation of academic research began around the same time Bayh-Dole legislation granted exclusive licensing of university patents developed with taxpayer funds to private interests (1980).
As an associate editor and frequent reviewer, I can say with confidence that peer review filters out at least 90% of the garbage that journals receive as submissions. It also provides useful feedback to researchers who cannot otherwise get it. People submit papers knowing they will get rejected because they want to read the referee reports.
In my field, academics simply do not trust a result until it has been replicated and scrutinized by other researchers. If that doesn't happen, it's usually because the result was not interesting enough for anyone to care. If the scrutiny reveals problems, that undermines the credibility of the authors, which is ultimately more important than whether they get the pub.
A final comment: Peer-reviewed journals differ dramatically in their quality. If you don't know what the good journals are, you should expect that your reading will include a bunch of crap that no serious person in the field would pay attention to. Why these crap journals even exist is a mystery to me.
Thank god for CS's culture of pre-prints on arXiv.
Promotion committees (hopefully!) know which journals or publications are low quality. If they are unfamiliar, they generally don't have time (or inclination) to verify the paper(s) or publication itself. Especially true when it's a slightly different sub-field than their own. So it often comes down to a simple count of pubs. That's usually all the credit they need!
For all the flaws of academic publishing, we are still in a much better place than when church dogma was the gatekeeper of knowledge.
There are some situations where peer review has led to groupthink. The one that comes to mind is the amyloid hypothesis. [1]
[0] https://theconversation.com/hate-the-peer-review-process-ein...
[1] https://www.science.org/content/article/potential-fabricatio...
> It has come to my attention that academics are now using generative AI (Chat GPT or whatever) to conduct their peer reviews.
> [..]
> If ever there were a compelling argument to totally abandon the intellectually dishonest notion of “blind” peer review – a system everyone knows is broken, rarely fully “blind” and frequently not “blind” at all, but allowing unscrupulous or mediocre yet established scholars to sabotage promising work – then this finally is it: accountability and transparency in the face of the robot takeover. Every person who reviews another person’s scholarship must be willing to sign their name to their own evaluation, to stand by it and assert it was not the work of the machines.
Unethical academics, AI, and peer review https://nicospage.eu/blog
If you used a machine to write your peer review, you’ve staked your reputation on its output being correct (in the same way that you stake your reputation on not producing bullshit peer reviews, which is to say… eh, probably not a make-or-break thing but it is in the mix). So you need to check the output. That’s the skill we as a society value.
We don’t employ scientists for their literature skills, but for their ability to build and evaluate theories and data, that sort of thing. And hey, less brainpower remembering grammar rules means more for the study of science. This is good.
Some people will torpedo their careers by putting too much faith in the ML model, but the way to avoid being them is to apply the same level of diligence to the model’s output as any other tool.
I’m more worried that ML models will enable some continued silliness. There’s something odd going on if people type their actual arguments into an ML model, then it produces extra necessary filler text to get into a journal, then we use ML models to review that text and hope to distill it to the actual arguments, and then the general public again uses ML models to do the same. It seems like we’d just be better off sharing the prompts or something, hahaha.
Empirically, I find no improvement in reproducibility between arxiv and journals. The costs are incredibly high too.
Like many things in our world peer review is a short lived extrapolation which doesn't resemble it's origins but is regarded as immutable gospel. It matters most if what you need is the respect of academics.
https://tvtropes.org/pmwiki/pmwiki.php/Main/NoHonorAmongThie...
Journalists are a huge problem here. I blame about half of the anti science backlash on breathless journalism writing checks science can’t cash.
This isn't a science-specific problem. Journalism has serious issues with incentives.
But what do you do without news?
By the time they are caught cheating, they have invested dozens of years if not decades into a career that is now pretty much dead. Is there a life-after-cheating story in the relevant field? I can't imagine much of one. Part-time lecturer/tutor in a fourth rate school, perhaps.
Of course, that presumes a moment in time where they begin cheating, risking it all. If they were cheating all along, from age 12 onward, maybe stopping is the problem.
I do not think peer review adds much value over self-publishing. Consumers still need to read and verify the work themselves. Bad stuff gets published in peer review so often you still need to verify the integrity of everything yourself. For a simple hypothetical, say peer review is "good quality" 80% of the time, and self-published is good quality 50% of the time. These are made up numbers, but I am saying "80% is too low for the pain of peer review to provide much value". As a consumer I find 0 value in peer review (I can use google and read what I want, being published in peer review is an annoying paywall if anything).
I believe the majority of comments in peer review are not based on technical accuracy (what an outsider may think peer review is about, verifying if something is right or wrong), but tend to be more clearly opinions. So from a writer standpoint for people who say "peer review improves my work", that does not jive with my experience.
There are so many other negatives with peer review in academia (people bean counting pubs, paywalled, the club issue Daniel mentions), I just don't think it adds much of any value. If everyone decided tomorrow "I am just going to publish stuff on ArXiv" (or whatever preprint server), the world would not be worse off. I think we would be better off actually.
But we are not an honor society anymore
sounds like the definiton of an echo chamber
The formalized system of opaque and unaccountable criticism that gatekeeps science had to have been invented in the 20th century.