Why even bother banning them. This just silences the opinion, which technically could still be valid. I'm sure at some points in history gay and civil rights activists would have been the ones getting banned.
I agree about the down-voting, but I truly feel we need to stop babying people in regard to controversial opinions. If you can't reject and ignore a blatantly racist and unfounded opinion, I'd argue you shouldn't be participating in discussions of any kind. This seems like a fundamental skill necessary to coexist in human society.
Edit:
I'm not saying you shouldn't be allowed to participate in discussion if you get angry and offended by an opinion. This is totally fine. However, if you need someone to step in and hold guns/ban hammers to peoples heads to exist, maybe the problem isn't with society, but with you.