The answer is everybody, in their own respective spaces.
It's not me but not others. It's me AND all others.
If I run a bar, I get to kick out customers who voice despicable opinions. Exact same thing applies if I run a shop, a web forum, or a conference for that matter.
All of us gets to kick out people who behave badly from our own private spaces, nothing obligates me to tolerate abuse.
Free speech does not mean freedom from consequences. It does not mean the right to an audience. It means you get to say your opinion without the government interfering - and then other people get to decide how to react to it.
Freedom of speech without the right to react and respond to what others say would a horrible idea. Partially because it entirely invalidates the reason for why speech is supposed to be free. If you can't change things based on what you learned from what you heard, why even bother letting people speak?
If this means somebody else bans me from their space for being offensive, then so be it.