As for intolerance, I'm not sure how we could consider your unwillingness to tolerate certain kinds of offices as anything other than intolerance.
The differences between what I said and what you said: mine could apply to any group, not just women, and I'm assuming that only a smaller subset of most people are willing to pursue high social status despite the extra burdens; most people are fine with middle status.
Also, intolerance of an environment is different than intolerance of a group of people.
Regarding middle status jobs, there are plenty of formerly male-dominated middle status jobs which women have successfully broken into, e.g. advertising (c.f. Mad Men).
Any explanation of why women do not enter math/phys/CS must come up with something that distinguishes math/phys/CS from those other fields. Being a "middle status job" doesn't do it, nor does initial unfriendliness. I'd suggest that one possible explanation is that "geeky" pursuits are actually lower status than most comparable office jobs, and more women than men are turned off by low status.
On the other hand, the idea of being repulsed by hanging out with low-status men… as a woman, that resonates with me.
The best thing that's happened for my happiness with programming is the arising of more well-rounded, socially skilled developers, who started off as music professors or linguists, like the ones I found when I switched to Ruby.