Peer Assessment Bias
ryansmulholland.com
ryansmulholland.com
I agree with this author’s sentiment that peers should not be dismissed in favor of experts. This is especially true when considering early stage speculation or anecdotes, which can be the seed of an eventual change in expert opinion. Expert communities can also develop their own subcultures and associated biases. We have seen this in academia in the US (strong leftward ideological bias) but it also applies to academic journals, which have traditionally been viewed as a record of expert knowledge. A recent example: the authors of the largest mask study have had trouble getting journals to publish it (https://www.theblaze.com/op-ed/horowitz-danish-newspaper-rev...). A former NYT reporter recently shared that the study’s author feels it’ll be published only as soon as a journal is “brave enough” to do so: https://mobile.twitter.com/AlexBerenson/status/1317875526997...
Blind faith and blind dismissal are both obstacles to seeking the truth. Unfortunately the side effect of the heavy politicization of everything is that every situation, even peer review or the publishing of scientific information, is a power struggle between ideologies, rather than a civil exchange of ideas and joint mission to ascertain the truth. Correcting this requires general awareness of our peer and expert biases.
Also see related past discussion about peer reviews as the gold standard: https://news.ycombinator.com/item?id=20607259
That is so wrong that I'm having a hard time finding where to begin.
First off, you are literally spouting off an existing political narrative which is intended to discredit decided science that opposes certain political viewpoints. IE, you shouldn't trust science because it's biased against your politics and conducted by people who are more concerned with not liking your political ideology than they are with propagating fact.
Have you ever gone through peer review or are you just spouting off something that appeals to your political sensibilities? I have published and peer review is the most formal and clinical procedure in the whole scientific process. It is usually conducted by three anonymous scientific rivals who have literally no incentive get your article through. Because of that, there are strict rules and norms about what is acceptable to say in these sorts of things (moderated by apolitical (I shouldn't even have to say that since this is science not politics) editors, plural for contentious topics). Any hint of political bias would get picked out immediately and the reviewer would get the boot. If you aren't satisfied then there is even a process to appealing to the editor's boss and so on. You're talking about like six strangers with competing interests that would all need to collude to force a political viewpoint into an article.
The peer review process is also just that: a process. The reviewers don't have ultimate power over you and if they make a bad faith claim for political reasons, then you get to call them out on it in your response. I actually know someone that was told he was too inexperienced to write an article (a pretty BS criticism). When he just wrote a terse reply telling the reviewer to not appeal to authority the editor accepted the paper and ignored the criticism. Imagine what would happen if you told the editor that they shouldn't accept the paper because it disagrees with politician so and so or something?
Even if your article gets rejected, there's typically like a dozen different journals that could equally well accept the article (but with less prestige possibly). It's not uncommon for articles that aren't about a politically contentious topic like masks to bounce from journal to journal until it's accepted on like the fifth submission. It isn't because of some sort of spooky political censorship, it's because sometimes there's legitimate criticism to be had with a manuscript.
If your article is so bad that you can't even get it included in lower ranked journal then nobody is stopping you from just releasing it. You can always dump it on a preprint server or even just host it from your own website and scientists can use it if they consider it acceptable. Nobody is censoring this information.
There is no such thing as an elite cabal of scientists who are suppressing political ideologies in peer review. This idea and the narrative that facts and science are something that are optional and have a political bias is IMO one of the most toxic ideas in today's politics.
Apologies if I’m being overly direct, but this claim seems to assume that groups of people who are part of the same institutions or ecosystems can’t share the same cultures or biases. It’s not a question of incentives or explicit coordination - it’s a matter of a lack of diversity of thought leading to individuals making similarly biased decisions or judgment calls even without coordination. These biases can also turn up in an insidious manner - for example scrutiny being applied in a discriminatory fashion (less for studies that are agreeable and more for ones that aren’t).
> This idea and the narrative that facts and science are something that are optional and have a political bias is IMO one of the most toxic ideas in today's politics.
I’m not saying that facts or science (in the purest sense) are optional, but I certainty think they are subject to political bias. Calling something a “fact” or labeling it as “scientific” doesn’t make it so. It isn’t a toxic idea to note that the practice of science can be biased even if the broader philosophy of science isn’t supposed to be.
I think of this as more of a human reality we have to confront if pursuing the truth matters. For example take a politically sensitive topic like whether there are gender pay gaps or not. Many researchers claim there is a gap based on univariate analysis and will affix their conclusions with labels like “facts” and “science” and “data”. Multivariate studies on the same topic often say there isn’t a gender pay gap or that sufficient data doesn’t exist to clearly make a case for it. These studies, however, are often not accepted by the broader public, not amplified by news media, and aren’t socially acceptable to even quote.
We don’t have to think very hard to search much to find examples of taboo topics that are off limits in academia/research, and we don’t have to work hard to find people who are the victim of cancel culture due to their scientific pursuits or statements either (example: https://www.thecollegefix.com/far-left-scholar-activists-att...). Likewise we don’t have to work hard to find gibberish accepted by the governing processes either (example: https://areomagazine.com/2018/10/02/academic-grievance-studi...). It is simply a plainly observable reality that human qualities corrupt scientific or scholarly processes that we want to have blind faith in. But the reality is we can’t afford to have blind faith in anything, not even scientific processes, and should apply scrutiny to what comes out of those processes just as we would to anything else.
The system worked here and it's manufactured outrage from the conservative tabloid you linked to over some random academic drama. The academic organization laughed off the letter (which in reality was just a google doc signed by 600 unverified people, most of whom aren't even claiming to be serious scholars, but are like random students on the internet looking for a laugh) and literally nothing happened. The news here amounts to "so-and-so wrote an angry letter." Not really worth the pixels it was printed on.
> Likewise we don’t have to work hard to find gibberish accepted by the governing processes either
I don't think you fully understand the nuance of the "Grievance studies affair." I don't blame you because up until this year when I heard a really good explanation of it I was also rooting for the authors. However, now that I fully understand what happened and why the papers were accepted I completely understand the ire of academics who were upset at the media reaction to it. If you have time, please listen to episode 70 "Doubling-blinding dog balls" of the podcast "Everything Hertz" (the podcast is about how to fix the reproducibility crisis).
To briefly summarize: the reason the journals accepted the articles wasn't because their fields are so poisoned by ideological extremism that they will accept any crap that confirms their pre-existing beliefs. That's the political narrative of certain reporting on what happened. What really happened is that the reviewers recommended to accept because the papers contained useful DATA despite their batshit crazy conclusions. In fact, the reviewers that have since spoken out did criticize the questionable interpretation and often gave constructive criticism to broaden it, but recommended to accept because of the data.
Take the case of the "dog rape" study. The body of the work was that the authors (fraudulently) claimed that they had collected like 1k hours worth of observational data where they sat in a dog park and compared owners reaction to male dogs humping female versus male dogs humping male dogs as they looked at different cultural factors. If people really do have a statistically significant difference in how they respond to their male dog humping another male dog versus humping a female dog, could that tell us something about how human male-male relationships are viewed by our culture from the lens of animals being anthropomorphized by their owners? It's not my research interest, but especially with 100k hours worth of observation, that sounds like it could be some really useful data to somebody out there and is worth publishing. I think that it's worth publishing even with an interpretation that you object to because an expert in the field can come to their own conclusions.
The true irony, however, of bringing it up while claiming to pro-"diversity of thought" is that you should be applauding the journals in this instance. They disagreed with the the wacky interpretations that the authors provided, but published anyways because they felt that it wasn't their place to change it. The same would happen for whatever your favorite right leaning topic is they had also lied about legitimate and useful data on that article. Diversity of thought is exactly what happened here and you can't and ask for voices that you interpret as politically left of center to be censored unless "diversity of thought" is really just code for "make science agree more with my ideologies."
Also, journals aren't some sort of "governing body" for science. Science is decentralized and there like a million independent journals out there that publish what they want. There's also the preprint servers I was talking about with basically no moderation (outside of plagiarism and blatant fraud). You can also just go ahead and write blog posts about your science and if its valuable, then people will listen. This happens in CS and ML research.
> I certainty think they are subject to political bias
They really are not and that conflation between political interpretations of data and serious analysis are part of the problem. Right or left, everybody can agree whether 2 + 2 = 4. Your reporting of the output of your instruments doesn't depend on who you vote for. Even in cases of the wage gap (I don't have a horse in this race and so I'm not going to dive into details about it) everyone can agree that "x-analysis" on "y-dataset" gives "z-output". You can even agree that "x-analysis" has "M and N" deep issues that make it unsuited for this type of data in a consistent way that doesn't depend on how political the data you are dealing with is. It isn't up to debate and science has nothing to do with what you believe is and is not amplified by the media. If experts aren't going for your favourite analysis technique, then it probably has to do with serious flaws in the method and I'm going to guess that the reason you like that statistical technique has more to do with getting results you agree based on your biases with than performing serious science.
Look closely at what you perceive as scrutiny being applied unevenly. I'd hazard a guess that these "facts" are exactly the ones being pushed by partisan media (not scientists) with an agenda to push. You probably are seeing scientists constantly disagreeing your media outlets because its exactly those outlets that are choosing deeply flawed analysis techniques to support their ideological opinions and then getting criticized for it. Its all projection. These pundit-types know that if they were in the place of these scientists and were studying a contentious topic that they would bend and distort their analysis to fit whatever narrative they are trying to push. Probably because that's literally their day job. Because they know that they would bend the truth, they just assume with no evidence that this is what's happening in science and then complain on their shows that their conservative voices are being censored.
You're not the victim of some sort of liberal conspiracy to warp reality to its will. The head of my research group is a deacon in the catholic church. I don't know his political opinions (because science is apolitical) but I'd hazard a guess that it leans more than a little bit to the right of center. There's no broad culture of suppressing your ideologies and the whole "conservative victimization" thing is a political narrative and is being pushed really hard right now because there's an election. Besides, if you really think there's a lack of conservative viewpoints in science then stop sitting around and complaining about it and just go do science. Nobody is stopping you and you can even publish on pre-print sites without affiliation.
There's a reason we trust experts. It's because they have worked on whatever problem you're trying to solve for like the past decade and have already discovered all of the wrong ways to attack it for you.