This, more than the jokes and comments, is the meat of the problem. Inappropriate jokes can be much more easily addressed as soon as they happen - if you're in a meeting and your boss asks you to 'please pull up your blouse because your wonderful breasts are distracting everyone' you can much more easily address it right then and there.
But if your boss asks you to take notes, the first couple of times it happens, it could just as well be random (though it isn't, really). It is much harder to say 'no, not taking notes, you only want me to do that because I'm a woman' even when it's true. What's worse is that even when you do notice a pattern, it's harder to address than a rude remark. It's (a) hard to prove it was because of your sex as opposed to some other aspect of your personality ('maybe he thinks you're just good at organizing potlucks, sheesh' - 'you took notes that first time so well!') and (b) behavior is much harder to correct when you have to point out things that happened in the past. 'I take notes 50% of the time, in a group of 5' just doesn't seem to have the same effect on humans, especially in a society where intent is often judged above effect.
I'm not a woman, but this is what it seems to me, from what I've observed.
edit: I can't find the study I was looking for, but they had a group of people evaluate two sets of identical resumes, with female and male names, for 'competence' and 'likeability'. For males, competence was correlated with likeability, but for females it was anticorrelated, even though the resumes were identical. Less people will think you an 'ice queen' if you call out an inappropriate remark, but countering the above form of sexism seems far more difficult to do while preserving 'likeability' -- 'what's the big deal, I just asked her to take notes!' If anyone else knows where the study is, I would be grateful.