I can't answer this question. That's my point, I have no clue why people care about what they perceive as sexual comments being directed at them. It just doesn't make sense to me.
Here is my thought process; if words can't hurt people physically, and I personally don't care about anything anyone says to me, there must be other people in this world that feel the same way. If this is the case than harassment must be on some level subjective, but a objective analysis of common reality would come to the same conclusion. It's a simple fact that what is sexual harassment in America today is not the same as it was 10 years ago, or the same as it is for a tribe in Africa, and I don't believe an objective definition of sexual harassment could ever exist. Therefore sexual harassment can not be universally banned in the workplace or otherwise. At least not in a non authoritarian rights based system.
Of course if all you care about is your perception and your subjective reality, and you believe it's ok to push this on people, my point is moot. I just lean towards basic rights and leaving the subjective stuff alone. It all depends on how you view reality. Personally I feel reality has a common base of extremely simple truths (science/normative ethics), and all else is essentially arbitrary. Is this right or wrong? I'm clearly not qualified to say, nor is anyone else.
Edit:
It's like trying to find an answer to the equation "2+2" if you don't know anything about math. Some people say the answer is between 14 and 20, some really strange people may say it's 2,000 and they know it for sure. Personally, I think it's best to say it's between negative and positive infinity and wait for additional information.
The question of the ethics of sexual harassment in the workplace is "2+2", and nobody on earth really know anything about "math" in this regard. So defer to the least restrictive answer.