> What if I'm one of those people, should I not be allowed to make public comment?

If you do, do not be surprised when the service you're using gains a reputation accordingly.

> And on the odd occasion I do, I just ban the user or the entire instance and move on.

Yeah, that's actually part of the problem. If anyone can ban anyone for any reason then you don't actually have a federation. You have, at best, a gathering of barely-interoperable fiefdoms. You can either have a federation of commoditized servers or you can ban people you don't like -- you cannot have both.