Banning people is an upside. Setting boundaries of acceptable behaviour is necessary for having a space where people won't be troll-brigaded and hate-swarmed, and where the worst people on the internet don't drive out everyone else.
Personally, I don't mind having to manually filter/ignore non-conformant behavior in order to prevent that possibility for myself and any other minority group online.