OP can host his own server and invite his friends and they can say whatever they want there.... but no one has any obligation to give him a venue.
If I have a shitty band and I want to book your venue for a gig, you have every right to say to me "no, your band is too shitty for my venue, I don't want to degrade myself or the public in that way."
There's very little nuance here at all, no one is saying he is not allowed to be a terrible person, they're saying they don't want to be associated with him because he's a terrible person.