So would you suggest we should have a safe space where people have to listen to each other?

I will defend people's right to say things I don't like hearing. However I will also defend other people's right to not listen (as long as they don't try to prevent the speech from happening) - that is also a freedom.

To modify a quote: Your liberty to speak ends where I am forced to listen to what you are saying, my freedom to not listen ends where I prevent you from speaking.

Downvoting does not prevent the speech from happening. It merely says I don't like this. Flagging does...