Who cares if I'm a minority or conservative? Who cares what speech I'm upset by? That's the point, being upset is not something we need to defend against.
Speech is speech, if you don't like it then ignore it. And yes, there are consequences. Most of the time it's the other side yelling back which is fine. Only when it turns into (harmful) action should be people be defended but until then it's paramount that anyone can say what they want.
I understand that only the government has protected speech. Private groups and institutions are free to invite or exclude at will and I support that as well, however public spaces and institutions should absolutely not be closed to anyone who wants to speak their mind and protecting people's feelings will never end well.