I'd argue that this is unreasonable for any project large enough to need a CoC at all. What it amounts to is excusing misconduct by project insiders, while allowing them to drive off new users by declaring their behavior 'annoying'. This isn't a code of conduct at all; it's a formalized "old boys club" dynamic.
In fact, a CoC is no more than them laying down the law explicitly on how they want people to behave in the project.
"If you are more annoying to work with than your contributions justify, you'll be ejected" is spot-on on how things work in general.
That might lead the project to fail or fork (beauty of OSS) but that's life.
There's a risk that this happens, sure, but it can be more charitably read as allowing/retaining people like Steve Jobs or Linus Torvalds, who, despite being assholes, are effective assholes and usually spew vitriol only in the interests of the product itself.
Weighing annoyance against contributions is a valid methodology. Your concern is valid, as well, so governance should always be introspective enough to ensure that the metrics of "annoying" and "useful" are objective ones and don't grandfather in the insiders.