When you let people spew hateful things you drive away the people you want in the community and are left with a toxic cesspool that no one wants to visit. Your moderators will burn out and leave as well. That's a very reliable way for your space to die.
Then there's the fact that it takes far more energy to refute bullshit than to spew it, and this asymmetry means that "just let them speak" means the toxic liars win.