That is the reason - from http://nodevember.org/statement.html : "that his presence would make some speakers uncomfortable to the point where they refused to attend or speak.". Instead of addressing this point, we generally are looking for concrete, binary, true and false examples. So it's harder for me to see how his past behaviour makes me uncomfortable. But I also can understand how it can possibly make other people uncomfortable.
How should any conference deal with speakers being uncomfortable with things?
How can comfort be increased for all? How can a small conference defend against accusations of uncomfortableness? What would the best way to do this be?
Personally, I don't think "well I'm comfortable, so it's okay" would be an acceptable response.
So, working towards an answer to my own questions, instead of looking for logical reasons, and responding coldly, we should look towards emotion, and respond in kind: "Please be welcome here, we will have separate speaker lounges, multiple places to interact and speak with each other. Many people from a wide range of backgrounds will be attending. It will be our priority to ensure you, and everyone can feel safe and welcome and have an enjoyable time at the conference. We are a group of local friends putting on a great conference about a topic we all like, we arn't a soulless corporate conference. We are approachable and thanks for trusting us and feeling like you can approach us with your concerns."
Respond with reassurance, comfort, trustworthiness, essentially more PR and less rationality (and I would suggest that if they did that, the speaker wouldn't need to have been removed)