And if you can only mute, you will not see it all but instead be a vessel to spread abuse to all your followers.
The town square analogy sucks. No one would accept a hate group going into their circle to shout slurs. Free speech absolutists don't care about other's freedom to live peacefully, only their own "right" to harass people.