I think if you manage an open source project you should feel comfortable taking a stand on contributors who have/advocate for actively discriminatory positions. Just because you organize a community project that may be used by people you fundamentally disagree with doesn't mean you have to check your sense of morality at the door.
If someone submitted a PR to my project who held openly racist positions, I would deny the PR. No amount of genius or technical merit makes up for that.
One could come up with a hypothetical of an open source project that is important enough to require any help it can get, and also somehow obscure enough that only a handful of developers are working on it. Nonetheless, I don't really think most open source projects (or even for-profit projects) fall into that category. At the point where work becomes so important that you need to compromise other parts of your character in order to finish it there are probably bigger things to worry about.