The argument isn't that he contributed nothing or that his contributions are worthless. Few people are advocating for burning everything he built to the ground simply to exorcise his lingering influence once he is gone or whatever and even fewer are doing so in good faith.

The argument is that by retaining his position he is actively alienating people. Do you think he is such a phenomenal genius that keeping him is worth ten good people quitting or not joining in the first place? A hundred? A thousand? Where is the break-even point where it becomes more beneficial to keep a "genius" than being welcoming to new people who feel deeply uncomfortable around his behavior. And are you measuring him by what he has created in the past, or are you really considering his present value and likelihood of future contributions?

What if he's not so brilliant after all and you're allowing one "genius" to cost you equivalent or even superior "geniuses" with less abrasive personalities?

We like to tell stories about individual heroes, but if there is one recurring pattern in history it's that we're always standing on the shoulders of giants -- and even that imagery is misleading as it is not really just a handful of giants but armies of them, mutually supporting each other as they carry your weight.