"We" are not using it for the same reason nazis are not using it: because hardly anyone's there. It needs critical mass first. Once that happens, nazis/trolls/etc will find their way there too, I'm afraid.
You can ban the nazi symbols & language and still have freedom of speech.
Twitter has a pretty well rounded TOS and they tend to go out of their way to block/ban people even when they haven't violated the TOS/CoC when there is sufficient pressure.
That said I don't understand why people want to turn their social networks to echo chambers, if you only having discussions that do not make you feel uncomfortable or challenge your views you often do not hold discussions worth having.
The mastodon.social terms ban this stuff:
> - Content illegal in Germany and/or France, such as holocaust denial or Nazi symbolism > - Conduct promoting the ideology of National Socialism
So, er, literally "Nazis", not just "things I don't like"
Then again, Twitter has been notoriously uninterested in fighting this kind of problem. In fact, they don't seem to care much at all what their users think (editing tweets, anyone?), probably because users are not a direct source of income for them (but advertisers are). This is different in the case of Mastodon. It would be interesting to see what kind of solution they might come up with.
Each of those instances are (potentially) run by different people, who can each have their own rules and limit entrance however they like, to ensure they can maintain their own rules. If an instance wants to allow nazis, or doesn't want to enforce any rules at all, then fine, but most others will block them. The nazis can have their own little corner of the internet, but nobody else will talk to them.
In practice, if Mastodon reached hundreds of millions of users, I'd expect that to be spread over a great many instances, run by many independent groups which makes the enforcement problem _mostly_ simpler. It's not perfect, but it's an interesting approach.