What are the downsides to code review? I don't buy that it is a velocity thing.
The only category of changes that I'm willing to accept that may not benefit from code review are small, trivial, low risk changes. (If you think large patches would not benefit from a review process, that's a deeper question but it something if feel 100% to be the case.) But, lucky for us, it turns out that small, trivial, low risk changes are also extremely fast to review. (Like, less than 5 minutes often.) And, haha, sometimes these 'trivial' changes turn out to be not so trivial once a second set of eyes lands on it!
The idea that "developers can take responsibility for their changes" is a tautology: if engineers knew up front that a review was unnecessary, then they would know what to expect in the review. But by definition, the point of review is to catch issues and get feedback that are inaccessible to the author. Therefore, it's impossible for an engineer to self-assess if a change requires review: it would mean they could predict the outcome of a review as well.
The amount of time a review takes wrt to velocity is a self-regulating function, in my experience. If you are dragged on by a review, it probably means the code warranted review. If the review is rapid, then the review was probably less necessary. But it was fast to do anyway, so doesn't meaningfully impact velocity.
Situations like critical bugs and fixes actually are even more important to review, even if they are small patches, because in a incident response situation people are more likely to cut corners, skip steps, or make mistakes due to stress.
So what exactly is the point of skipping code review, given that regardless of patch size, there are clear benefits? The benefits are large enough that I consider the burden of proof to be on those who suggest code review be an optional step for deploying changes.
The one exception I will raise are externally enforced deadlines, where review is not possible in time due to mitigating factors affecting the reviewer. (For example, the reviewer is a domain expert but is out sick, and the software has a necessary delivery date.) But even in these cases, the review should happen in-full post-deployment, and it should be acknowledged that this is a risk-laden move. And often these are showing secondary flaws in the process (why do we have only one domain expert who could review this? why didn't we have more time allocated for review? etc)
In practice, I think a lot of aversion to reviews originate from the frustration of thinking you are finished with a piece of work but having those expectations have to shift after review raises criticism. Everyone wants to ship, especially when they themselves have checked the box that 'it works and I made it as correct as I know how to do.' Addressing reviewer feedback is fundamentally one where you have to draw upon the higher goal of improving code quality and improving your own skills, and like most things that cause real improvements, it can hurt. Honestly, I don't think anyone is free of guilt of feeling this way at one time or another.
Beyond that, other fundamental problems can lead to an aversion to code review, such as reviewers missing the forest for the trees, causing reviews to devolve into pedantry, reviewees misjudging the value of others' feedback, mistrust between team members wrt their feedback being valuable, external pressures due to misaligned expectations about delivery, etc. Counterintuitively, strong aversion to code review, an engineering task, can sometimes stem from all kinds of messy, complex problems with team dynamics. So keep in mind that if people are continually complaining about code reviews, this might be telling you something much deeper going on!
I think in general a solid code review from a peer you trust is one of these things that generally speaking, hurts in the way pushing yourself in exercise does. I personally feel it fosters a kind of skill growth you cannot get otherwise, except perhaps through deliberate practice. In other words, no pain, no gain.