Your second paragraph seems to assume a fixed number of developers working on a project, and in that case there is no downside in replacing them. But what about an open-source project where they keep sending PRs for bug fixes and features that all have technical merit? Would you ignore that and redo the work? If I'm working on the same project, should I avoid merging their work to accommodate you?
While I agree that a community where everyone can feel comfortable is something to strive for, I think that does not require excluding some members for the views they hold. It is enough when they don't push those views onto the others.