None of the guys I'm around, nor myself, think that's okay just because women aren't present.
i would posit that you're not getting as whole a picture (definitely of men, possibly of women) as you could if nobody you know does.
Well, we aren't stupid either, and I'm sure most of us didn't take it that way, so if you want to address the contextual nature of his comment then maybe do that instead of scandalizing yourself in pity for the silent nongeneralizers.