I realized I wrote a wall of text in response. I'm not picking on you, but I think you brought up a point that I think is really worth responding to in depth, because I felt that way in one point in time, and I've slowly realized how wrong I was.
I used to believe that I would surely notice if I was in an environment that fostered harassment. Over time, I've realized that harassment is a problem, but I was just looking for the wrong things. Harassment doesn't really manifest itself in a way that can be made clear from a single objective measurement.
There is a lot of research about sexism that really reveals what is not obvious to me, as a man, about how sexism affects women:
When talking on the phone in a customer support role, women are more likely to be yelled at.
Men tend to talk for a disproportionately large amount of a conversation.
Assertive women are viewed much more negatively than assertive men by both men and women.
Studies like these made me re-evaluate how harassment presents itself. I used to think that it was obvious, like wolf-whistles on the street. Harassment is more like living in a world where everyone is just a little (or a lot) bit shittier to you. It's a lot easier to ignore, because you've had the same experiences.
As an example, I currently am one of three guys out of 25 people at my current company. We are a non-profit, and we have about 800k constituents, 100k of whom give in any given year. We all field a lot of calls from constituents who aren't happy for whatever reason. In the two years I've worked here, I've been yelled at once. However, the women who work here get yelled at weekly, regardless of their position.
I majored in biology, and took a ton of CS in school. Bio was mostly women, and the CS program had barely a handful. As I look back, I've realized that the women in the programs really needed to present a lot more evidence for any other their proposals than men ever did.
In both cases, it's really easy to dismiss criticisms with the platitude "Those things happen to everyone." They do, but they happen more to women. These little things add up, and teach women that their point of view is easily discounted, even when the evidence is incontrovertible. Reporting sexual harassment, even to someone you trust, is a huge gamble. All of their life experience points to the fact that they won't be taken seriously, especially if the person they are reporting it to is a male. They are way more likely to be seen as not being a team-player for reporting it, even if they are 100% in the right.
Long story short: in order to see harassment, it's incredibly important to listen to what women are saying. It's also really important to listen to what you are saying in response, and whether you are actually listening.