We should not confuse perpetrators and victims.
From the several hundreds of men I know as friends, class mates, colleagues, I have never experienced a single one cat calling, groping, grooming or doing worse kinds of sexual harassment. In hundreds of parties and thousands of work situations, I've never seen it. Not even once.
I've seen both men and women circulating sexually charged jokes or telling such jokes in work environments, and I've seen both men and women comment on each others looks or hitting on each other, both when it was wanted and unwanted. But that's it.
Yes, that's just my anecdotes, but it seems far more likely to me that we are talking about a small percentage of men doing it all the time than most men doing it sometimes.