You argued against all of the above on the other side of the coin. But the moment its an issue that doesn't affect you, you're quick to resort to the arguments you're advocating against?
That "PC Minefield" is avoiding that "someone, somewhere will think I'm a sexist". Which is the problem you admit to existing but don't care about. You'll care when you're the one being seen as a sexist with a Twitter hate-mob at your employers' heel demanding you be fired for an offhand comment or action that wasn't even sexist unless twisted and taken out of context.
It's not about "growing thicker skin" it's about avoiding HR. If I don't speak to them or deal with them, there can be no case for harassment. I consider it job security.