The difference is the societal context. Women are judged by and valued for their appearance (to the exclusion of other attributes) to an extent that men are not.
The things women talk about in regards to men would put even the manliest of construction workers whistling after every woman on the street to shame.