Hacker News Reader

Comment by jedberg | Hacker News Reader

It blows my mind that peer reviews aren't done blind. Not only is there this geneder bias but also bias of "I know that person they're famous I'm sure their work is good" or "I've rejected their paper in the past".

Seems like an easy fix would be to make all peer reviews blind.

In fact, for any study that gets federal funding, they should have to publish their hypothesis ahead of time into an escrow system, submit their paper to the same system, and then get blind peer reviews (blind in both directions) where the reviewer gets to see only the initial hypothesis and the paper with the names/institutions removed.

And of course all the papers should be available for free. Maybe the government could pay reviewers directly for their time, but I haven't thought that one through yet.

View on HN

> It blows my mind that peer reviews aren't done blind.

I've co-authored a few papers and blind reviews have some surprising consequences, like discussions along the lines of "you can't say that you already wrote about this issue elsewhere and put a link in the paper, because that would unblind it". I was a bit uncomfortable with that, because I like to "cite my sources" (even if the source was myself in this case).

This also points to another issue: The more specialized the paper is the less likely the blinding will work. If you know a field by heart then you know who works on what, and probably can guess most authors based on that.

I'm not saying I'm against blind review, but while it sounds obvious, it has some issues in practice.

Whoever told you that about your cites is probably right -- you shouldn't cite yourself while calling out that you are citing yourself. You should cite yourself the same you that site any other work.

The second part is probably unavoidable. If you're working on something super specialized when all your reviewers are the six other people who work on it then sure, it'll be unblinded. Not much we can do there sadly.

I disagree, the style to write as a "neutral" observer (often writing in passive voice) is frowned upon nowadays for good reason.

Part of that is also that the information if the authors wrote a citation or not can be important to a reviewer. For example it is unfortunately quite common that authors publish results in a salami tactic to maximize the number of publications. There can be a significant difference in impact between a citation saying "this is important to work on" which is written by the authors and one which is written by someone else.

Generally we should not write to hide information, and that would include if the authors wrote other work that relates to the work being reviewed. We should not adjust our writing to doubleblind review (and I would argue the advice the author was given is wrong). Doubleblind review is imperfect anyway, I can often tell who the authors are just by topic and e.g. writing and figure styles, so if a reviewer really wants to know the authors they can. We should still do double blind though.

> the style to write as a "neutral" observer (often writing in passive voice) is frowned upon nowadays for good reason.

Could you be more specific about who's frowning upon it? Because I've never heard this before in my field (Comp. Ling., where double blind is the rule) and would like to look more into it.

Many style guides now say to write in active voice (Nature is one of them, but many others as well). I don't have the books in front of me, so can't find the citation, but many publications on scientific writing essentially recommend direct language.

The reasoning is that the work was "subjective", i.e. carried out by you. By using "detached third person language" you are trying to give a false impression of objectivity. This is similar to management/PR double speak like "we are forced to raise our prices", "we are unable to compensate you"... (I don't assign malice in the case of scientists though).

Like sibling I'd also like more information on this and can't find anything immediately compelling with a light search. Like this post suggests, obviously there's a use to hiding information, sometimes a bad reputation is undeserved or irrelevant to the work, sometimes there are subtle biases in play.

The reputation of the author and their behaviour wrt the citations used should be considered, I agree with you that it's important information. Maybe the reviewer of an individual paper shouldn't be considering that though, maybe that should primarily be considered in a second review stage or in the context of meta-reviews? Idk, but the spirit of using passive voice in the context of research makes more sense to me.

> For example it is unfortunately quite common that authors publish results in a salami tactic to maximize the number of publications

Why is this unfortunate? I'd argue that splitting results in multiple publications is a) Riskier for authors (higher chance of rejection) and b) More convenient for readers (each paper requires less mental load, being focused on a single aspect). So, even if there's a payoff for authors, it doesn't come for free.

The tactic is more advantageous to authors, because they get more articles (which is used as a metric to evaluate scientific success) and I would argue it's less risky. Say you split up your results up into 3 papers, your chance of one paper being rejected might be higher, however your chance of all papers being rejected is lower.

In terms of more convenient for readers, you discount the mental load required in finding papers. That's in fact one of the biggest problems in many scientific fields at the moment. There are so many papers being published that it is very hard to keep up with the field. Reading the same amount of results also requires a much higher load, because if authors split up the results into 3 papers, the individual papers are not suddenly shorter, but in fact the overall page count is typically almost 3 times of a paper that would have put everything into a single paper.

Writing about your own work in third person has nothing to do with using the passive voice. You can use the active voice just fine: "Papers X,Y,Z show that ..." works regardless of the identity of the authors of X,Y,Z So the style argument is wrong.

The other argument you have is also questionable. It doesn't matter at all the identity of the citer when using citations to argue that a topic of research is important. If you cite 20 papers and they are mostly from the very same author, it doesn't matter if you're the author: any reviewer will realize the claim is shaky -- or not, if all those papers happens to be actually outstanding.

Double blind is imperfect but miles better than single blind. And we shouldn't list made-up defects that don't stand to scrutiny to it.

> Writing about your own work in third person has nothing to do with using the passive voice. You can use the active voice just fine: "Papers X,Y,Z show that ..." works regardless of the identity of the authors of X,Y,Z So the style argument is wrong.

I agree that writing in third person about your work is not the same as writing in passive voice. It is part of the same style trying to give an impression of objectivity despite the fact that you did the work.

Essentially you are trying to hide the information that you authored the papers, and did the work. Just compare "In paper X,Y,Z the authors show the importance of proper citing" or "In paper X,Y,Z we show the importance of proper citing". Don't tell me that you would not evaluate the 2 sentences differently.

> The other argument you have is also questionable. It doesn't matter at all the identity of the citer when using citations to argue that a topic of research is important. If you cite 20 papers and they are mostly from the very same author, it doesn't matter if you're the author: any reviewer will realize the claim is shaky -- or not, if all those papers happens to be actually outstanding.

Sure if there are 20 citations it's very obviously shaky, but often enough cases are not quite so clear cut. I still believe one should not deliberately hide information.

> Double blind is imperfect but miles better than single blind. And we shouldn't list made-up defects that don't stand to scrutiny to it.

Just so I don't get misunderstood, I'm not arguing against double blind, we should always do it and I have been advocating for it in several settings. I just say we should not suddenly change the way we write papers, so not to accidentally reveal our identity to the reviewers. That approach will make papers more difficult to read and write with questionable benefit.

In some cases the issue that you can't really "blind" it because it is a physical continuation of the earlier work - not just building upon the idea, which anyone could do, but using the exact same experimental device (a unique one, like the hadron collider) or continued analysis of the exact same set of patients or animals, or improving the previously made software tool.

This happens in larger fields too. For example in AI, which is a large research field, you can be blind-reviewing a paper, but it uses a proprietary google dataset, so you know where it comes from. Blind review is not the answer, I fear.

For me the most important advantage of blind review is not that 100% of papers are effectively blind, but that if you're a noname author you have the right to be blind and not having your paper looked down on just for that reason. That alone justifies double-blind review.

Regarding papers that cite proprietary datasets that no one can access, in fields like AI where it is perfectly possible to release datasets (if there's a specific reason it's a different issue), as far as I'm concerned they should be outright rejected due to lack of reproducibility and inability of the reviewers to check the correctness of the claims. Although I know this is a minority viewpoint and it won't happen.

Another factor in my field as well is linking to software -- papers that introduce algorithms or use computational analyses are rightfully expected to include the code (usually github) and reviewers are expected to at least check that the code exists and is reasonably documented. I can remove my name from the paper, but the github page will say which lab it's coming from.

Academics just like every other aspect of human life is full of politics and this includes peer reviews as well. Papers regularly get panned for variety of reasons that have nothing to do with actual content of the paper and maybe even worse sometimes because the reviewer happen to be working on something similar and want secure a bigger grant for themselves.

That's false. Trust the science.

"Trust the science" is an oxymoron. Please do substantiate why grandparent's claim is false. There is a massive literature on the shortcomings and, yes, even potential for malice, of current peer-review practices.

https://sigbed.org/2022/08/22/the-toxic-culture-of-rejection...

https://www.nature.com/nature-index/news-blog/research-misco...

The sarcasm went way over your head lol

The user you're replying to is reminding us we just endured ~2 years Faucism.

Peer reviews are usually blind (authors don’t know who their reviewers are). What you are referring to is double blind (reviewers don’t know who the authors are). The challenge with double blind is anonymizing content that often relies on cited prior work.

> The challenge with double blind is anonymizing content that often relies on cited prior work.

You're the second person to bring this up, but I'm not sure why it's a problem. Just don't say "based on my own previous work" and instead say "based on previous work". I.e. cite yourself the same way that you'd cite anyone else's work.

If the previous work is a single paper, you are right. However it consists usually in multiple papers, it becomes then more complicated to hide that all these papers share one (or more) author(s).

But as a (final) reader I would weigh supporting evidence by the same author/author group/lab far weaker than supporting evidence by unrelated sources! That being said: One could get around that by replacing the citation style for the review version only. There are typically changes anyways and the given amount of fights that I had with editors for pure typesetting or "improvements of english" that one would actually be helpful to do.

As the final reader you're expected to do some due diligence and at the bare minimum read the reference list. Once you do, you can easily see which papers are self citations and which ones are not. If you're not doing this due diligence you probably don't care too much about this paper anyway and there is no harm done if you think a reference is not a self citation when it actually is.

That doesn't help if references are given as First Author et al. or even as [number] and while I agree in principle it simply doesn't happen in practice.

But then it just weakens your statement to make it worse. And I'm not even sure if it makes a difference for what the article talks about. Because the reviewer needs to actively go looking. And at least to my understanding these effects are not due to people going out of their way to misjudge people, but rather that it's an effect of subconscious prejudices. And for the latter breaking the obvious connection is probably enough.

Speaking from personal experience, it can be really difficult hard to write a paper that anonymizes yourself...especially if you are known for a particular approach to a specialized problem. You actually have to begin excluding cites, turning them into placeholders (withheld due to anonymous review requirements), rather than just speaking about the work as if someone else did it.

I like where you come from, but a LOT of phds are so narrow in scope that its almost impossible to be anonymous because the set of peer reviewers is very small. Only the large wide fields like medicine and some variants of humaniora are big enough to do this.

If you're working on something super specialized when all your reviewers are the six other people who work on it then sure, it'll be unblinded. Not much we can do there sadly. But most people aren't working on something that specialized, and most would probably benefit from having reviewers in adjacent fields and not just the other experts in their field. So at least those adjacent ones would maybe be anonymous.

Interesting, I hadn't thought of that. I wonder if that's okay though — if you're that familiar with someone and their work I'd expect your bias towards that specific person overrides any broad gender bias.

As a rule I'm generally not a huge fan of taxation, but I'd be happy to see my tax dollars go towards a system like you propose here.

I recently commented on a similar discussion here about a month ago[1] to the effect that double blind review is a cheap and idealistic shortcut that runs into its own issues (research areas are simply too specific to pretend that we're operating "blind") and doesn't get you what you really need which is to address the deeper issues in the publication system and research system.

These issues? Off the top of my head:

1. Quantity over quality

2. Normalised sensationalising of ones research

3. Neglecting good or even necessary collective scientific practices such as replication studies and data and code sharing and openness.

    a. Here, valuing the actual work of peer review comes in as a fundamental aspect of the scientific process that should be respected, rewarded and published rather than being some silent aristocratic duty that eventually gets navigated around in the sensationalism rat race many researchers pursue.

4. Selecting for "impact" and "novelty" and not the quality of a researcher's/scientist's administrative and leadership skills, scientific method and integrity and teaching skills (incl, importantly, the teaching of graduate students)

     a. Though novelty and impact are important in research, IMO, they're outcomes that are hard/impossible to select for largely because research breakthroughs are often serendipitous and the kind of true genius that will "hit targets no one else can see" is rare and frankly everyone knows it when they see it (provided they're good researchers and not just good salespeople).  A very senior academic once told me in a private context that you can't predict where the breakthroughs are going to come from as an inexperienced researcher playing around is just as likely to make a breakthrough as a senior researcher with many credentials.

5. A personal belief of mine ... resisting the necessary professionalisation of research, by which I mean that the conception of a researcher is based very much in the same model we had centuries ago. IE, a lone genius researcher left to their own devices to find the truth. Scientific research, at least, is now too complex, too hard and involved for this to be true. Collaboration and more and more specific roles are necessary to make the industry work well. The resistance of the "industry" to recognising the importance of software developers in research is a perfect example. Same goes for statisticians and consultants from adjacent fields. My personal favourite for such an "associative" role would be quasi-theoreticians (outside of physics) who can aid in aggregating, reviewing and critiquing the literature in real time without necessarily having a horse in the race.

~~~

[1] https://news.ycombinator.com/item?id=32832836

Reply on news.ycombinator.com