Authors’ names have ‘astonishing’ influence on peer reviewers: study
nature.com
nature.com
We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials.
We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's sports beforehand (women's basketball), or no sports. And finally we randomized whether the person giving the subjects instructions would be a young woman (one of my classmates) or a young man (me).
Our small study showed what you'd expect- students- men and women students rated male writers more highly than female writers, and initials were right in the middle, though they tended to be more like responses for men.
The tester's gender made no difference that we found.
The sports thing made a measurable difference, but it didn't reverse the skew.
My 18/19 year old self was somewhat skeptical we'd find a difference, but I was totally wrong. It taught me a lot about bias and perception, which this study also shows.
I imagine these biases swing in all sorts of directions depending on the context. Some are intuitive, many are not.
Also pay-gap has likely also to do with the current men-bias also? Wonder if that would be the same if we had 90% woman throughout in these decising positions..
In the end all mine assumptions, what I actually want to say is: I don't think those two contradict or relate at all.. what GP already said well with "biases swing in all sorts of directions depending on the context"
From your wikipedia link
> There are two distinct numbers regarding the pay gap: non-adjusted versus adjusted pay gap. The latter typically takes into account differences in hours worked, occupations chosen, education and job experience.[1] In the United States, for example, the non-adjusted average woman's annual salary is 79% of the average man's salary, compared to 95% for the adjusted average salary.[2][3][4][5]
The remaining 5% could be from the "women are wonderful" positive attributes not being the narrow selection of ones that are highly sought after in well-renumerated jobs, or (spitballing here) psychosocial (Expectations of a pay gap driving negotiation behavior or something).
Even at the time, there were studies that showed traits associated with women were rated more positively by men and women than traits associated with men.
The way you study this is you'd give a list of words:
gentle caring assertive stubborn aggressive loving
(ideally you'd randomize the word order too)
And then you'd have a group of subjects rate them as more associated with men or women on a scale.
Ideally you'd get a big sample and replicate this study.
Then either in the same study at a different time, or another study, you'd take those same words and you'd ask your subjects to rate them as positive or negative.
That's where you see effects like the one mentioned in the wikipedia article.
That kind of thing's been done a bunch of times.
But studies on gender and competency have been done too, and at the time they showed the same pattern as we found.
I say "At the time" because this is >20 years ago.
Right. There are two different sets of benefits that help or hurt someone in different ways. Being competent when standing trial can work against you while being wonderful will reduce the risk of a conviction and reduce the sentence if you are convicted.
We have identified the "competent" bias and are taking steps to correct it, but we need to do the same with the "wonderful" bias in other systems. For starters we need to recognize how strong that bias is in certain fields. For one example, there are specific crimes that people would bet are extremely gendered in nature, and the crime statistics show they would be right, but interviewing the population at large and querying victims, including those who never went to the police or who were turned away by the police (or even worse, who couldn't legally be victims because of how biased even the laws are), we see the gender component goes away. The rate of men victimized by women and women victimized by men are at near a 50/50 ration (I think 49.8 to 50.2).
Even the extent of studies measuring the impact of the wonderful effect is lacking compared to studies measuring the competent effect (which itself is likely a bias of the wonderful effect).
If there is a bias, is it good, bad, or neutral? Are there biases in other directions?
Bias isn’t inherently good or bad - they are normative judgements of distance from rationality.
Some biases are probably evolutionary advantageous to the individual but harmful to the society or others.
Sticking with the group is great, except when it isn’t, for example.
Or, a “gut check” is great, but you can’t always make decisions based on your gut:
I think if it had been a larger study (not a one credit class run by 4 undergrads but an actual study with funding) we might have had multiple texts and randomized them, but we had one text- I don't even remember what the topic was.
They told one group of Asian women that “women are worse at math than men” and another group of Asian women that “Asians are better at math than non-Asians”. Both are common stereotypes.
They then measured how well the two groups did on subsequent math exercises.
Interestingly, they found that the latter group (positive stereotype) performed better than those in the former group (negative stereotype).
As I recall, other research found similar results with Black males and golf scores when told “white people are better at golf” as opposed to “black people are better athletes”.
Not only do stereotypes influence our perceptions of “the other”, but they influence the performance of the other.
”You call me an addict and refuse prescription? Yeah sure whatever doc, I’ll just buy that fentanyl off the street since I’m already an addict.”
We need to end prohibition.
[1] https://www.tandfonline.com/doi/full/10.1080/23743603.2018.1...
I wish people would stop bringing up studies that don’t reproduce, they’re no better than anecdotes.
Are they still a reputable source, today?
Setting this aside for a deeper read, but it appears at first glance that the concerns revolve around statistical technique rather than methodological soundness?
How much more? That's really important.
Most things I read I have no idea of the sex of the author. Do people even look at author names before they start reading (online, say), and even then they can be nom de plumes, or non-gendered names (sometimes surprisingly).
During debriefing, we would tell the subjects what we were actually looking for (perceptions based on gender) and most reported not remembering/caring about the gender of the author, but nonetheless the results were clear that there was a gender bias, regardless of whether they reported remembering or caring about the gender of the author.
At the time, I felt like "Sexism is dead" and the university I was in was 2/3rds women to 1/3rd men, so we'd never find a bias.
As a young man (18/19) I learned to question my assumptions. I thought "It's the 90s, sexism is dead.", but it just wasn't true.
The simpler version of the study had been done before, and the class I was a study design class. Our class assignment was to design, execute, and analyze a study.
We were simply combining two existing studies into a new study.
It bears mentioning- this was ~1996/1997 (I don't remember), I was a freshman, it was a single credit supplement, and we never published the results.
I don't see this "bias" as inefficient or counter-productive. It is just an artifact of the way you designed your flawed experiment.
To find any real bias you would have to assure the subjects that the researchers were equally accomplished.
What do you mean? I think you didn't understand the experiment. Subjects were shown the same text with, at random, a male name, female name, or initials. These names were not real researchers, nor familiar to the subjects.
> If it is known that most discoveries are made by males it is reasonable to give larger weight to new male-done research
I'm sure you can see the two different fallacies in this short sentence:
- There have been more male researchers than female, therefore more "discoveries" were made by men than women. No shit. This does not mean male researchers are better than female, obviously.
- There are more men being researchers therefore men's research should be given more weight therefore there are more men researchers therefore...
1. most discoveries are made by males
2. it is reasonable to give larger weight to new female-done research
contradicting you point.
No it isn’t. Dont you see this is just circular reasoning?
I think it's safe to say that this study can be treated as an anecdote because we don't know the effect size or exact methodology and it was never peer-reviewed. And I'd agree with your point if we were talking about the Nobel prize study, which evaluates people as individuals. But arguing that 'real bias' is bias that doesn't come from empirical evidence doesn't make a whole lot of sense- all bias comes from empirical evidence of varying quality.
Just one of the issues is the specific perception regarding that particular sport (basketball) and relative perceptions between sex differentiated similar sports (men’s and women’s basketball are not the same sport, albeit similar). A sport should have been chosen where there are as few differentiating perceptions as possible, which is nearly impossible, because men are inherently more competent at sport due to physiological realities. It bakes in a bias towards men, which is what you likely actually confirmed with the study you described.
Introducing sports alone essentially corrupted that research and likely biased the responses for relative innate competency of males in sports.
For example, ignoring its own challenges, what could have been done, is show video of a relatively broadly positively evaluated sport (not basketball), an activity that is also relatively broadly and positively evaluated female clustered competencies, e.g., effective child rearing, communicating effectively, conflict resolution, creative outputs, or even beauty pageants, etc., along with the inverse, i.e., poor performance of men in the same sport and poor performance of women in whatever is chosen, e.g., screaming and yelling at poorly behaved children.
It is in the past and by no means are or were you the only one who has long engaged in this type of poorly executed “research” that has merely been confirming researcher biases and often, thereby immensely damaging society, but maybe my illustration of a few of the several issues with what you described, will cause some changes in thinking.
There is a real hidden epidemic of not only single order thinking, but what really should be called negative order thinking in research, where research is not only not finding the truth, but rather even doing damage through confidence in false findings.
I say these things with an expensive relevant background, in the trenches of bad and … frankly … destructive research, if you will.
- when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication
- when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications didn't really add much to the scientific knowledge, but they didn't even have the very high standards required when I was in Italy
Why is this bad? You get more funding the more you publish...So labs that publish easier due to the names involved get much more money...which they can use to publish even more and get even more money...which they can use to publish even more...
There's excellent scientists anywhere, really. But funding, fame and politics are extremely asymmetric in academia.
In my discipline, most journals even have double blind peer review. I thought that's the standard.
It's fundamentally impossible because when you publish the topics are going to be in a way or another niche enough that you know all of the people working on those topics and conferences make those circles even more public.
Say that you are studying e.g. "helical peptides", which is a vague and broad term on one side, on the other one, all the people that work on specific aspects of helical peptides (say, using them to bind surfaces that are very different in polarization, e.g. batteries) know each other and blind review would add nothing really. Even when it is blind you likely know who's reviewing it most of the time.
There are topics that are of course wider.
To make a comparison with programming: imagine you're involved in an open source community with a language that is niche enough that you know most of the people involved in it (it is common for many lisp dialects).
You can likely be given a program in that language and figure out likely who wrote it due to their interests, programming or api designing, or the use of static types, or some kind of linting that you know only a handful of people use, background, etc.
With the huge difference that in case of science the circles are much smaller because you know all your peers are in academia, who of them has specific instruments (not everyone has the same lab equipment), but who doesn't also have some others, you may know that some of your peers have their niche and "main branches" of research so you figure those things much easier. Also, in those circles you also know who is involved in your "code reviews" so if you know that A and B aren't reviewing it (you know they would've told you) it can only be C, D and E, but the feedback requesting this kind of experiment could only come from "C"...
Even if you know the people in the field and you can guess the authors, double blind does not hurt. Actually, I don’t see any downside of double blind.
Actually, names are very often used to let you out of the field if you are not part of the usual family…
There are many systems we use which provide anonymity, and they always result in some form of abuse. Such as phone numbers (scam calls), Internet (cyber harassment, scams, viruses), cryptocurrency (all the scams). Any system where humans can gain advantages from abusing anonymity they do so.
I'm not an academic so maybe there are already safe guards, but from my understanding people already cut any research into as many small papers as they can. Truly double blind peer review would likely encourage this. Maybe it would also further encourage people to steal research or peer review outside of their expertise.
The benefits may outweigh the negatives, but I'm sure people will find a way to abuse it too, consciously or not.
1. If a reviewer recognizes the author without declining the review, the result cannot be worse than without blind review. A difference would only be if the names of the reviewers are also not anonymous, but this creates much worse problems and would seriously jeopardize the review process.
2. If a reviewer doesn't recognize a dupe or paper very similar to one already published, then the reviewer is not competent and the journal has a problem with the reviewer pool or reviewer selection. That is a problem in any case, and is the reason why bad quality journals exist.
3. If the reviewer is incompetent, unfair or even insulting, then the author will complain to the editorial office who will then investigate and possibly get a third reviewer. The editor in chief or area editor can see the reviewer replies, just not the names of the reviewers and authors.
4. Your point about stealing research is not an issue. The publication is not anonymous; theft will be discovered very quickly by the scientific community (which will discover it more likely than two reviewers anyway). However, I'm pretty sure many top journals also use anti-plagiarism software.
I really don't see how double blind review could be abused more than non-blind review.
Two examples from last year: * https://www.google.com/url?sa=t&source=web&rct=j&url=https:/... * https://www.universitetsavisa.no/etikk-forskningsetikk-forsk...
How so?
For smaller papers, if the papers still stand up to review even when the research has been cut up, that seems a neutral effect at worse. It might even be seen as a benefit as each paper is more focused.
As for stealing results, the names will still be attached before the printing is done. Only the review process will be double blind. Thus any ability to detect stolen research will still occur.
On the one hand are famous scientists who have wide influence, on the other hand are small academic circles working on obscure, niche topics where everyone knows each other so well that blind review is impractical.
How can one scientist be both so reknowned that their name tilts the balance of grant money unfairly away from other less known researchers, and at the same time their research is so specific that only a very few others can even comment on their work?
An analogy may be how heavy readers can immediately tell which author of their particular topic area they are reading, based on a short excerpt. It is what makes alternative noms de plumes almost impossible, especially now with systematic and documenting AI.
Even some Ivy League universities are now requiring blinded applications for faculty positions and they have double-blind reviews to avoid bias, e.g. hiring a graduate from Oxford instead of no-name university.
If I walk two corridors over - still in my department, but just a slightly different aspect of the field - they won't even recognize the name.
Often the arXiv upload will be right around the time of a conference deadline in fact. For these people I see no problem with an OpenReview-esque setup where the arXiv page remains anonymous until the authors opt to reveal themselves.
If people don't care for true double blind review then they should stop pretending they do. Conferences like ICLR and NeurIPS are far from blind. It would be one thing if it were just a matter of trusting reviewers to not Google the paper title, but it goes beyond that. There are currently no restrictions on the use of social media to advertise works under review nor on the timing of arXiv uploads, which can trigger alerts to those in the field right around review time.
Reviewers prioritize their time above all, and after that correctness，leaving novelty in last place. If a paper comes from a unknown lab it tends to get greater scrutiny because a famous name is a subconscious stand-in for correctness. You spend less time reviewing famous authors because you think they are less likely to have bugs.
Reviewing a very good paper is quick and easy ("LGTM"). Really terrible papers are not too bad if they're obviously awful, but reviewing something subtly flawed takes a lot of work: you need to identify the flaws and describe them in a way that's compelling enough to convince the authors--or at least the editors. Ideally, you'll also explain how to address them, which is more work now and down the road when you review the authors' often-grudging implementation of your suggestion.
In such a world, people may be increasingly reluctant to review papers without some indication of their quality (for which name is a rough proxy). My solution to this is to somehow make reviewing more valued. It's an important part of science and deserves more than a checkbox.
Reviewing is hard work with very few rewards - and you have to decide at the start whether it is worth your time to embark on it. Ideally we would like to learn something from the paper.
I and most people here would most likely strongly prefer to review a paper written by someone good at what they do.
In reality most papers are pretty bad - thus when we prefer reviewing papers written by Nobel prize winners we are not "biased" and "humbly bowing" before their greatness - we are just trying to make it worth our while.
So I think most computer scientists would agree with the article's conclusion that double-anonymous reviewing, while flawed, is better than the alternatives. I don't think I've done a non-anonymous submission in 10 years, and as a reviewer, usually I don't have a strong guess about who wrote the paper (and when I think I know, often I turn out to be wrong). It's a little annoying that this news article in Nature Magazine ignores the longstanding widespread prevalence of this practice in a conference-driven (but... we like to think important) academic discipline. :-)
But: (1) I don't think the big benefit of double-anonymous reviewing is that a lousy paper from a Nobel laureate (or, from CMU/Berkeley/MIT/Stanford) isn't let in unfairly. To me the big benefit seems to be that reviewers have to review every paper as if it might be from their friends or a famous person, and consequently have to review each paper with due care and try sincerely to understand its contribution, which maybe they wouldn't do if they knew it's from some random place/author they've never heard of or don't think highly of. I think the way it affects judgment may be more in equalizing time/effort spent to read and understand a paper (and the generosity you give a paper because "maybe" it was written by your friend or somebody you respect), rather than a straight-up bias towards liking whatever the faculty at a famous university are writing about this year.
(2) While I do think it's true, and a good thing, that double-anonymous reviewing helps "marginalized groups of authors who often struggle to have their work see the world" as the lead researcher says, we should probably acknowledge that authors are not the only beneficiaries of a scientific publication. The interests of the reader matter too -- the journal or conference has some duty to serve them. On the margin, maybe some readers would be more interested to learn what Albert Einstein is thinking about these days, or would like to see a well-balanced conference program that includes a good talk by a known-provocative speaker, instead of one more random (but adequate!) paper from a nobody. I'm not saying we should give a huge weight to this -- it's fine to make people eat their vegetables, but, I don't think we should act like scientific publication is only to give authors a line on their CV and the readers' preference is 100% irrelevant. A scientific journal shouldn't exclusively serve the authors. (Other kinds of media care way too much about what the reader wants, e.g. Facebook giving you whatever it thinks will keep you clicking things on Facebook, but there is probably a happy medium somewhere.)
(3) The challenging frontier may be in grant submission and reviewing, where proposers are typically not anonymous to the reviewers, which surely leads to some biases. I have heard about government programs where they did use double-anonymous reviewing and it seemed weird to me. (Probably this is a situation where track record should matter, yet trying to summarize your own track record while remaining effectively anonymous seems really hard...)
I’ve often seen celebrity authors write a “Letter to the Journal”, for those kind of readers, and I think that that might be the safer way for big names to be recognized.
It may give the author space to collaborate with interested parties without necessarily overwhelming the peer-reviewed content.
At the same time, some authors are truly just prolific.
(You might also reasonably believe that the job of an editor or program committee is to assemble the most edifying program or issue when considered as an ensemble, rather than each paper individually.)
I also don't think it's completely crazy to consider to a tiny degree what the reader "wants," separate from their true interest in an omniscient sense. When you click around on the Internet, do you only read the "best papers"? Probably not. This is similar to why the New York Times (and Hacker News) don't use 90% of their word count to remind readers to eat healthy, stop smoking, maintain friendships and regular activity, sleep regular hours, and give away most of their money to buy malaria nets.
Or some would rather read a mediocre Tweet from somebody they follow than a good tweet from somebody they don't know.
That's why OP calls it "eating your vegetables", forcing people to read the "best" articles rather than the articles they "want" to read.
But if that same paper in support of a fringe theory is reviewed under an anonymous name, it might not get published, even if it's perfectly well-argued, simply because "the topic is fringe and not very relevant to our readers" (circular reasoning).
And even if it's not fringe, how we view an author may influence whether their paper passes peer review for perfectly legitimate reasons. If a random scientist is the 100th one to weigh in on any old debate (let's say Penrose-Lucas), it may get dismissed as non-notable. But Penrose giving his updated thoughts would be notable.
There's costs to anonymous peer reviews. Credibility and notability can't be fully detached from the author.
> I don't think we should act like scientific publication is only to give authors a line on their CV and the readers' preference is 100% irrelevant
In many fields, the reader can go and read what they want in arXiv or a similar repository. And in the fields where this is not the case, it should be. The elite authors that you mention, in particular, shouldn't have any problem to get their papers read by linking them in social networks, starting a blog, etc.
While this wasn't the case 50 years ago, right now almost no one reads journals from front to back, people just search for individual papers, and the main purpose of the peer-review process of conferences and journals is basically gatekeeping and providing some signal for career evaluation, i.e., to "give authors a line on their CV". Thus, I don't think there is any reason to judge anything but the paper contents.
Seems like an easy fix would be to make all peer reviews blind.
In fact, for any study that gets federal funding, they should have to publish their hypothesis ahead of time into an escrow system, submit their paper to the same system, and then get blind peer reviews (blind in both directions) where the reviewer gets to see only the initial hypothesis and the paper with the names/institutions removed.
And of course all the papers should be available for free. Maybe the government could pay reviewers directly for their time, but I haven't thought that one through yet.
I've co-authored a few papers and blind reviews have some surprising consequences, like discussions along the lines of "you can't say that you already wrote about this issue elsewhere and put a link in the paper, because that would unblind it". I was a bit uncomfortable with that, because I like to "cite my sources" (even if the source was myself in this case).
This also points to another issue: The more specialized the paper is the less likely the blinding will work. If you know a field by heart then you know who works on what, and probably can guess most authors based on that.
I'm not saying I'm against blind review, but while it sounds obvious, it has some issues in practice.
The second part is probably unavoidable. If you're working on something super specialized when all your reviewers are the six other people who work on it then sure, it'll be unblinded. Not much we can do there sadly.
Part of that is also that the information if the authors wrote a citation or not can be important to a reviewer. For example it is unfortunately quite common that authors publish results in a salami tactic to maximize the number of publications. There can be a significant difference in impact between a citation saying "this is important to work on" which is written by the authors and one which is written by someone else.
Generally we should not write to hide information, and that would include if the authors wrote other work that relates to the work being reviewed. We should not adjust our writing to doubleblind review (and I would argue the advice the author was given is wrong). Doubleblind review is imperfect anyway, I can often tell who the authors are just by topic and e.g. writing and figure styles, so if a reviewer really wants to know the authors they can. We should still do double blind though.
Could you be more specific about who's frowning upon it? Because I've never heard this before in my field (Comp. Ling., where double blind is the rule) and would like to look more into it.
The reasoning is that the work was "subjective", i.e. carried out by you. By using "detached third person language" you are trying to give a false impression of objectivity. This is similar to management/PR double speak like "we are forced to raise our prices", "we are unable to compensate you"... (I don't assign malice in the case of scientists though).
The reputation of the author and their behaviour wrt the citations used should be considered, I agree with you that it's important information. Maybe the reviewer of an individual paper shouldn't be considering that though, maybe that should primarily be considered in a second review stage or in the context of meta-reviews? Idk, but the spirit of using passive voice in the context of research makes more sense to me.
Why is this unfortunate? I'd argue that splitting results in multiple publications is a) Riskier for authors (higher chance of rejection) and b) More convenient for readers (each paper requires less mental load, being focused on a single aspect). So, even if there's a payoff for authors, it doesn't come for free.
In terms of more convenient for readers, you discount the mental load required in finding papers. That's in fact one of the biggest problems in many scientific fields at the moment. There are so many papers being published that it is very hard to keep up with the field. Reading the same amount of results also requires a much higher load, because if authors split up the results into 3 papers, the individual papers are not suddenly shorter, but in fact the overall page count is typically almost 3 times of a paper that would have put everything into a single paper.
The other argument you have is also questionable. It doesn't matter at all the identity of the citer when using citations to argue that a topic of research is important. If you cite 20 papers and they are mostly from the very same author, it doesn't matter if you're the author: any reviewer will realize the claim is shaky -- or not, if all those papers happens to be actually outstanding.
Double blind is imperfect but miles better than single blind. And we shouldn't list made-up defects that don't stand to scrutiny to it.
I agree that writing in third person about your work is not the same as writing in passive voice. It is part of the same style trying to give an impression of objectivity despite the fact that you did the work.
Essentially you are trying to hide the information that you authored the papers, and did the work. Just compare "In paper X,Y,Z the authors show the importance of proper citing" or "In paper X,Y,Z we show the importance of proper citing". Don't tell me that you would not evaluate the 2 sentences differently.
> The other argument you have is also questionable. It doesn't matter at all the identity of the citer when using citations to argue that a topic of research is important. If you cite 20 papers and they are mostly from the very same author, it doesn't matter if you're the author: any reviewer will realize the claim is shaky -- or not, if all those papers happens to be actually outstanding.
Sure if there are 20 citations it's very obviously shaky, but often enough cases are not quite so clear cut. I still believe one should not deliberately hide information.
> Double blind is imperfect but miles better than single blind. And we shouldn't list made-up defects that don't stand to scrutiny to it.
Just so I don't get misunderstood, I'm not arguing against double blind, we should always do it and I have been advocating for it in several settings. I just say we should not suddenly change the way we write papers, so not to accidentally reveal our identity to the reviewers. That approach will make papers more difficult to read and write with questionable benefit.
Regarding papers that cite proprietary datasets that no one can access, in fields like AI where it is perfectly possible to release datasets (if there's a specific reason it's a different issue), as far as I'm concerned they should be outright rejected due to lack of reproducibility and inability of the reviewers to check the correctness of the claims. Although I know this is a minority viewpoint and it won't happen.
You're the second person to bring this up, but I'm not sure why it's a problem. Just don't say "based on my own previous work" and instead say "based on previous work". I.e. cite yourself the same way that you'd cite anyone else's work.
But then it just weakens your statement to make it worse. And I'm not even sure if it makes a difference for what the article talks about. Because the reviewer needs to actively go looking. And at least to my understanding these effects are not due to people going out of their way to misjudge people, but rather that it's an effect of subconscious prejudices. And for the latter breaking the obvious connection is probably enough.
These issues? Off the top of my head:
1. Quantity over quality
2. Normalised sensationalising of ones research
3. Neglecting good or even necessary collective scientific practices such as replication studies and data and code sharing and openness.
a. Here, valuing the actual work of peer review comes in as a fundamental aspect of the scientific process that should be respected, rewarded and published rather than being some silent aristocratic duty that eventually gets navigated around in the sensationalism rat race many researchers pursue.
a. Though novelty and impact are important in research, IMO, they're outcomes that are hard/impossible to select for largely because research breakthroughs are often serendipitous and the kind of true genius that will "hit targets no one else can see" is rare and frankly everyone knows it when they see it (provided they're good researchers and not just good salespeople). A very senior academic once told me in a private context that you can't predict where the breakthroughs are going to come from as an inexperienced researcher playing around is just as likely to make a breakthrough as a senior researcher with many credentials.
~~~
https://sigbed.org/2022/08/22/the-toxic-culture-of-rejection...
https://www.nature.com/nature-index/news-blog/research-misco...
The user you're replying to is reminding us we just endured ~2 years Faucism.
Non-sense. However, for all the people that say "trust the science" this is the science you're working with. It's politics all the way down just like everything else. For example, if Knuth decided to post some utter garbage chances are it would accepted based on his name alone.
What makes something political? If new research shows that X product is dangerous and republicans, who that company donates to, come out and disagree with the research they just made it political. Now, someone like you, can just say "This research is political"
You've created a way to dismiss scientific research simply by disagreeing with it
Science is a process and it is never complete. You are confusing a process with the results of the process, which are by definition imperfect. What makes Science good is that it's results are fundamentally imperfect and tainted, and that it respects that.
When you go around saying 'science is perfect' you disrespect its core principles.
It's possible to both have a generally high amount of faith in 'the science' while still remaining critical of its flaws, or rather, the flaws of the system in which its conducted.
If there's a vaccine and the majority of people in that field say it's safe but you want me to be critical what do I do?
I hesitate to use the word faith at all, even when describing the scientific process itself. Having faith in the process leads to (for lack of a better phrase) a failure to trust, but verify. IMO, it is one thing to approach scientific literature with a trusting demeanor and still walk through the proper processes to verify the results, and another thing entirely to literally take the author's word. Often times especially when big names are attached we aren't trusting-and-verifying we are simply taking their word. That's a problem that weakens the credibility of science.
I don't want to come off as someone who disregards everything he doesn't like. However, I do approach science with great pessimism and have since graduate school. Not because I hate it, or something is wrong with the scientific method, but because approaching it pessimistically allowed me to write better work by forcing me to actually perform the method instead of p-hacking my way to a result.
Rather, Knuth could publish some mediocre paper that would be instantly accepted by all the top journals, widely read, discussed and cited instead of more salient works on similar topics.
That's the problem really, excellence is rare while mediocrity is aplenty and academia is a numbers game. The mediocres with prodigious output will get the cites, tenure, TAs, lab assistants and budgets to do research and perhaps stumble onto some other mediocre result, reinforcing the cycle.
Emperor's new clothes, while the rest need to be whiter than white and all that. These things happen because people are blinded by titles in a complex world with complex people.
You mean like Elon Musk? When he came up with that giant box to extract kids from a cave with passages so narrow rescuers couldn't wear their air tanks on them, they had to push them ahead of themselves, reddit and HN and twitter were full of people defending him. There were even people defending him when he accused the rescue leader of being a pedo - simply because he was british and living in thailand.
Or how about the CEO of Brave, who apparently is an expert in public health, viruses, vaccines, epidemiology, etc? Endlessly defended here on HN any time his name comes up.
Smart people like Bill, Elon and the CEO of Brave are polymaths and can cross domains quite easily.
Musk is just a guy getting rich via insider trading. Gates is most likely using his foundation to push his interests.
But you should also remember that "science" doesn't guarantee that _every_ paper is good. No one can. "90% of everything is crap" is a law of the universe as solid as the second principle of thermodynamics.
Science is a _process_ that is able to weed out the crap more efficiently than anything else we tried. Any big name can put out crap, true, but as soon as they do, there is an eager army of people who jumps at the opportunity of "proving X wrong". If the original paper was crap, that would be a pretty easy thing to do.
Note that I said "more efficiently", not "efficiently" in general. There will always be an army of people "trusting the science" (the authority, really) and so it might take time for the crap to be weeded out. But it will eventually, because science is like evolution: if nothing can be built on top of the crap, it will be progressively abandoned, because it can't reproduce (pun intended).
That's correct and I agree. Actual Science doesn't require trust. As you aptly pointed out it's excellent at weeding out crap most of the time. The issue is the academic politics involved in it make it harder to determine if the process is weakened. The problem of course comes when a major result was not reviewed appropriately and it becomes the standard until someone is brave enough to write another paper failing to reproduce it. That's the scary thing - even if someone is brave enough to write a paper saying its non-reproducible they still have to get through the review committee to have their voice heard. Sure, they could post to Arxiv or a blog if all else fails...but none of that will ever get picked up where it is needed most.
> There will always be an army of people "trusting the science" (the authority, really)
This is really the crux of it. It's appeal to authority. Even higher order thinking people (that is, those outside pop-sci nonsense) fall victim to this because it's human nature. The authority is also how mediocre work gets past reviewers by attaching a name to it.
One thing I quickly noticed was that the articles were never signed by the authors directly. I found this striking, but also refreshing. It caused me to pay more attention to what was said. And, as I later found out, that seems to be the main intention[1]:
The main reason for anonymity, however, is a belief that what is written is more important than who writes it. In the words of Geoffrey Crowther, our editor from 1938 to 1956, anonymity keeps the editor "not the master but the servant of something far greater than himself…it gives to the paper an astonishing momentum of thought and principle."
[1]: https://www.economist.com/the-economist-explains/2013/09/04/...
Opening NRK right now gives "Now the government must see the madness in this", "Electricity prices can make going to the cinema more expensive", and "He is so tired" as the top articles for today.
I follow so much International/American news thanks to publications like Reuters, AP, and Qz. But no matter how much time or money I use, I can not follow my own country's news. I definitely am not alone in this, and it terrifies me how unworried most of Norway seems to be about this fact as well.
Its a ticking time bomb.
https://www.wired.com/2012/07/mklopez-digg-power-user-interv...
Reddit and IMDb are mostly immune. Which is not to say that they're immune to all problems (astroturfing, brigading, personal biases, etc.).
The main change is you can post posts to your profile rather than a particular subreddit. But that's not a big feature. Back in the day people used to just create a subreddit of their username.
They skim, see if it looks okayish and give it their approval.
But it's far worse, even without a recognized names, most votes are cast without proper reading, and I'm fairly certain also by the least intelligent subsection given how often submissions are upvoted on H.N. and Reddit that are pure clickbait and demolished in the comments by people that actually read it. People that vote by and large only read the title or the first sentence and make up their mind from there.
Agreed. I was referring to Quora.
Each group trended towards catapulting a single band / musician, but it always just depended on which one in the group got the momentum first.
I wish I could find the study again but it's hard to google for.
Pretty eye opening.
Academia was bought and paid for long ago and the money was used to build an incredibly broken and overly political bureaucratic engine of scholarly and scientific work that doesn't get anywhere near as much peer-review scrutiny as it should and commands far more respect in politics and legal proceedings than we should allow.
Universities and experts are the best we can do sometimes so we have to rely on it, but it doesn't mean it's truth or absolute and people like to use it as if it is to sell ideas like global warming instead of educating people on climate change.
This was my impression of most of my professors years ago too, especially as we saw so many quit for high paying engineering jobs at companies year to year — the ones who stayed aren’t in it for the money
I suspect it’s the same at the top: most senior administrators I bet would make more in senior Fortune 500 jobs
Once you take money from someone (except for NIST, in my experience) you're basically beholden to try your hardest to get the results they're looking for. Some scientists are moral enough to still return bad results. A lot of scientists aren't. There's a lot of garbage out there, and the worse the journal, the more garbage it gets. Famously, Phillip Morris studies "passed" the scrutiny of several major journals. It's amazing what greasing a few palms will get you.
I disagee, you should backup your accusations or statements (unless widly accepted) with some reputable source. This person claimed that science was bought and paid for. Did they mean all of it? Or most? That's an insane accusation that requires evidence.
"Go to your favorite major journal and start looking at the "conflicts of interest" and "grants" section. Once you take money from someone (except for NIST, in my experience) you're basically beholden to try your hardest to get the results they're looking for"
This doesn't mean people falsely data. You're simply providing motive.
Everyone wants money, you are using greed to then claim mass fraud in science
This isn’t only about greed either. People want their research published for reasons other than greed. For example, they want to move up in their career or achieve recognition.
After looking at a lot of medical studies related to COVID during the last couple years, I have seen first hand how biased and inaccurate many of them are. Some of these studies are even mentioned in major news outlet despite their obvious flaws when you actually begin to scrutinize them. Think big pharma providing research grants for studies that conclude their products are effective.
The OP never said that people falsify data as a result of receiving grants from interested parties. They often don’t have to. They can simply design the experiment in a way that doesn’t account for specific variables or behaviors then use the resulting data to reach a specific conclusion.
I remember seeing an article related to AI research on HN a little while ago that somewhat explained this problem. The grant money all goes to people researching deep neural networks which creates a reinforcing feedback loop. Since all the money goes to one branch of research, it creates very few opportunities to research competing ideas. I believe it was this one:
Most declare no conflicts of interest. One author of one paper seems to have started a company based on similar technology: potentially a bias, but also potentially putting one's money where their mouth is. One other author lists some consulting work for a few companies.
As for grants, I doubt people are bending their results to appease the NSF or NIH. There's certainly groupthink in what gets funded. We're still throwing money down the ABeta-for-Alzhemier's hole, for example. That eventually shapes what topics get published, but maybe not the specific results. The recent Abeta articles are pretty negative, for example.
However, I think your climate change example is a bit strange. If anything, it's big oil who's been trying to sell the idea that we don't have to stop using fossile fuels. Hiding evidence, and spreading confusion by paying lobbyists and scientists. Global warming was proved beyond reasonable doubt decades ago.
Any questioning it, even slightly means being banned from grants and academia.
Its also interesting that most climate models are NOT open source. Most recorded data from satellites is also NOT open source. So everyone works with a pre cleaned data set.
Its also worth pointing out that data sets like HadCRUT have never been audited by any respected scientist/group of scientists. This data was collected in stations not meant for long term measurements and they have a lot of errors. Just download it yourself and see. (Climate scientists are not really data experts, since they go from clean datasets in school to "clean" data sets in real life)
Calculating global temperature is also one of those things that is done in quite an obscure way, extrapolating too much IMO.
No other hypotheses about geophysics are required to show realistic and fault free simulations of the climate of the planet over several centuries to be generally accepted. Why should we apply such extreme prejudice to the hypothesis of climate change caused by the greenhouse effect? The basic mechanisms is quite simple and well understood, there is a variety of kinds of measurements supporting the claim that the temperature of the planet is increasing (meteorological temperature measurements, glaciers disappearing etc.). Full understanding of all the geophysical processes and feedback loops involved is not necessary, and very likely impossible.
I also have a hard time understanding the motives for such an enormous scam. Who would stand to gain from this except a relatively small number of researchers and the renewable energy industry? On the other hand, it's well documented that the fossile fuel industry has tried to sabotage climate science for the purpose of limiting political action on the issue.
This is false. NASA and ESA science data is free.
There is often an embargo period for very novel sensors, and always a delay of hours-to-weeks to allow processing to catch up, but it's free.
If it's the source code of the analysis pipeline you mean -- even though you said data - that's a harder lift, because the processing is complex. But even that is changing (https://science.nasa.gov/open-science-overview).
Even in the absence of the open science initiative above, today you can always get the raw data ("Level 1 radiances") or sometimes even uncalibrated straight-off-the-sensor data ("Level 0"), if you want to process it. (https://www.earthdata.nasa.gov/engage/open-data-services-and... -- "All EOS instruments must have Level 1 Standard Data Products (SDPs)")
And if you want to look in to how the processing works, there are detailed documents ("ATBD's") that explain how the pipeline works, for each data product. Also free.
> Climate scientists are not really data experts, ....
Dreadfully wrong. Do you work in this area at all?
This does explain some of the recent and embarrassing [lack of] retractions of Nature papers though.
Reviewers are generally overworked (for other reasons), and the main goal is to be constructive and somehow not sound like an idiot (in front of the other reviewers, who might be subject experts and definitely know who you are!) in any of the pile of reviews you need to write after reading a pile of papers.
Also, reviewers declare conflicts ahead of paper assignments, which makes accidents less frequent.
In any case, this is the tip of the iceberg. The entire structure surrounding academic publication is absolutely ill-designed and few people in that world are even remotely interesting in safeguarding veracity.
The overwhelming majority of peer reviewed scientific results is infotainment for other scientists with no party having any material stake in the veracity of it, which is almost no one bothers to replicate because accuracy is irrelevant so long it be interesting to read.
The moment a company has bet a sizable stake on it's accuracy, then they suddenly check, and double check and have another party independently verify it because they do not want to loose money, obviously. But most science is nothing like that.
Authors regularly cite work by themselves or their team, so a statement like “In a previous study[1] we established a relationship between X and Y” renders double blind pointless.
That's why anonymous grading - and in scientific publishing, double-blind peer review is so important. It's part of the scientific progress just as much as replication, the attempt to re-produce results of a study post-publication by other groups (I wish papers' PDFs had a QR code to a Web page that said "double-blind review by x people, replicated by y groups" - the latter changes over time so it's better tracked externally).
Sometimes it seems very obvious that a given paper is by specific authors (either because of style, or because of how familiar it is with their previous work) but I've had many experiences where I later learned that my supposition was completely wrong. Similarly, when you encounter a paper that doesn't have any obvious cues (which is the overwhelming majority of them) then it's pretty much impossible to tell whether it's an author you admire or someone you've never heard of -- and this is a good thing.
Some conferences don't use blind submissions, and yes: I have felt an awful lot of influence there. "Surely [famous Turing-award winning authors] don't need me to double-check their proof."
I'm not sure how to fix this problem though. In our phage newsletter we try to avoid using names and universities to focus on the paper/topic/finding itself, but I keep finding myself looking at author names and affiliations before diving into any paper.
I know it's "wrong" and I recognize myself doing it, but I still do it all the time.
This is a good initiative, but a catch is that if he's the only person deliberately spelling some words British and others American, the spelling choice becomes a unique identifier.
Though, it could work as long as no one within the group knows who is using the varied spelling.
People should copy the writing style of famous auctor's then to expose the system and keep people honest because then they know there are copycats.
Besides, invited speakers are a thing for a reason.
(Also, it is a major red flag to cite 10 papers from one group and no related work from anyone else. Either the topic is completely irrelevant, or you didn't do a cursory literature search - at least skim the references from papers you cite!)
Double blind helps the former, though not the latter.
In a part of my field that is capital intensive, some well funded newcomers have recently invested a lot, and "broke in", while some incumbent testbeds went away; the "expensive equipment" problem is usually temporary if the field is expanding.
Alternatively, when IceCube 2 comes out, the old IceCube crowd might be focusing on other stuff, and not paying attention to the IceCube 2 politics. That makes them great peer reviewers (no horse in the race, but knowledgeable).
As in, that would be the null hypothesis. It would be astonishing if most academics overcame it.
EDIT: apparently, the size of the effect (factor of six more likely to be accepted if it came from a Nobel Prizewinner) is larger than anticipated.
Great feedback, she thought, because She was xxxxx.
First, the article talks about "double-blind reviewing" as being "logistically hard" because reviewers can "find the paper in a Google search", and talks about a "price tag". By contrast, many conferences around me implement so-called "lightweight double-blind" reviewing, which takes zero effort and just means that the authors and affiliations are not mentioned on the copy of the paper that reviewers read. I think this form of double-blind is a no-brainer, eliminating some bias with essentially no downside; and that discussions about the cost and complexity of double-blind reviewing are a distraction from this immediate improvement.
Here is a typical paragraph from a call for papers (here, STACS 2023) describing the policy:
As in the previous two years, STACS 2023 will employ a lightweight double-blind reviewing process: submissions should not reveal the identity of the authors in any way. The purpose of the double-blind reviewing is to help PC members and external reviewers come to an initial judgment about the paper without bias, not to make it impossible for them to discover the authors if they were to try. Nothing should be done in the name of anonymity that weakens the submission or makes the job of reviewing the paper more difficult. In particular, important references should not be omitted or anonymized. In addition, authors should feel free to disseminate their ideas or draft versions of their paper as they normally would. For example, authors may post drafts of their papers on the web, submit them to arXiv, and give talks on their research ideas.
It does not mean that reviewers lower their bar when assigned a paper from a famous professor, they just know a priori that the work will not be completely scam.
In practice, we may get better outcomes if peer reviewers completely ignore these aspects and evaluate all papers purely based on what’s on the page. But I don’t think it is obvious that any reliance on trust and reputation should be derided as bias to be eliminated.
Where to start here? What do we want the purpose of publishing and peer review to be? Out of all this publicity dance, which part gets distilled into the solid foundation of science? I do think this whole journals/publishing/conference/apply-for-funding thingy is a bit too ancient and incremental, and more radical solutions would be nice to try. I think fundamental to this is the structure of funding, do we really want small funds going to individual small PIs, or maybe more independence, or less independence ...
In my field they seem to draw from a very limited set of reviewers (often senior academics who have not worked in the specific field in quite a while). We have been criticised with very outdated information. This is worse because editors are not experts, we had a reviewer contradict textbook established science and when we asked for an additional reviewer, they send it back to the same person.
Even worse, their goal is not publishing good science, they want to sell journals. While they will never publicly admit it, I know that they take the into account the reputation of an author in their decision to send it out to reviewers (for those who don't know, in the high impact journals like science or nature, the biggest hurdle is typically getting the editors accept the paper and send the paper out to review).
> In 2008, in response to a series of Call-for-Paper e-mails, SCIgen was used to generate a false scientific paper titled Towards the Simulation of E-Commerce, using "Herbert Schlangemann" as the author. The article was accepted at the 2008 International Conference on Computer Science and Software Engineering (CSSE 2008), co-sponsored by the IEEE, to be held in Wuhan, China, and the author was invited to be a session chair on grounds of his fictional Curriculum Vitae.
It sounds like a related bias of automatic thinking here. There's some cognitive ease in recognizing the author, so your bias kicks in that this is better than if you didn't recognize the author.
For example, having done work in the past that was the subject of a Nobel Prize might have an "astonishing" influence. Having the name "James" would not not likely have an astonishing influence.
So, essentially it can be viewed as a diversity problem in the reviewer space. The journals have important responsibility on this and they can not present as neutral observers of this phenomenon where they say : "oh every peer reviewing model has problems " and blame reviewers ethos that they are choosing.
I’m no expert, so please correct if I’m wrong. But for example, how likely is that a Nobel prize winner produces worth-publishing research? Or how likely he is simply good at the skill of paper writing?
That being said an author isn’t published randomly in Nature, so I expect subsequent papers from an author to be better, on average, than non-Nature published authors.
Lifehack discovered: legally change your name to that of a Nobel prizewinner if pursuing academia.
Jobseekers with Anglo-Saxon, easy to pronounce and common names are the most likely to get to the interview stage compared to candidates with unfamiliar names, according to research by the Australian National University published in the Oxford Bulletin of Economics and Statistics.
https://www.independent.co.uk/news/business/news/unusual-nam...
"No no no, within which historical context does it even make sense to ask this question? No? You see?"
“Jobseekers with Anglo-Saxon” means “candidates who speak Anglo-Saxon.”
And I’m like “I thought that was just a medieval studies thing, but I guess the older languages really do help …”
As in this person has more than proved himself, let’s not vet him as much.
Right up to the very end, this man was able to publish like a celebrity because no on questioned it because he was famous. So let's not bake that into the process: scrutinize all papers equally. Experts don't get a free pass, if they have new claims to make, those claims are just as "I don't believe you yet" as anyone else's.
A famous name getting the same paper published easier is a failure of peer review. The whole point of science is that ideas and evidence stands on its own merit. Not celebrity or seniority or power or any other axis that doesn't matter.
You don’t want to go on reputation.
And your reputation follows you. If it’s a big-name lab who has an amazing track record, you’re going to review the paper in the context of their entire body of work.
(he had only one peer anonymous reviewed paper and it had an error)
". Albert Einstein only had one anonymous peer review in his career — and the paper was rejected2. This happened in 1936."
How is just one paper submitted anonymously being rejected an indication of a trend?
Duh
I'm not saying there aren't people in academia who aren't driven by doing good research but I certainly don't see that as the driving force in the US academia.
I didn't realize that HN was an academic journal where all statements should come with proof.
Do you have proof that I'm wrong?
i had a friend who changed their name in their CV and got 90% success rate, and was close to 10% before
My comment was snarky but not remotely playing the man. I meant it quite literally as a criticism of the thinking at issue. Economistic thought is riddled with confusion, and being surprised that scientists don't behave normatively (ie. not as those engaged in peer review are "supposed to") exhibits perfectly one variant of said confusion.
I'm not 'astonished' that non-blinded peer reviewers are influenced by social status. No-one I know would be surprised at all, let alone 'astonished'.
I am astonished, in the same way I would be astonished to find out that papers that smell like fish are more likely to be accepted for publication at the majority of peer reviewed venues, and that the reason is that reviewers for those publications let their house cats stack rank the submissions.
People will choose what they listen, see, or read heavily based on who the performer is.
A performer who has consistently given good content will obviously have a bigger pull than a nobody still trying to get their first break.
The identical paper being rejected more often based on the celebrity of the author isn't that.
I've seen it from the inside. Probably read 80-100 widely-cited papers during my PhD (before dropping out), and maybe half a dozen of them were written by people who had any interest in discovering truth and pushing mankind forward.
Seriously cannot overstate both the willful ignorance of established scientists, and the extent to which this is enforced onto the next generation.