A good part of it is that regular people don't want to deal with them. They don't want to hear from them, and they definitely don't want to be harassed by them.

Look at the "free speech" platforms, the *Chans. They may not have started out as bastions for neo-nazis and racists, but they became that because, as hate speech and such was not really policed on there, normal people stopped going altogether, leaving only those people behind.