This is a good point.
I am a married man. As a woman, my wife cannot give me guy time - hence I love to hang out with my friends (who are all guys) - and I cannot give her girl time - hence she loves to hang out with her friends (who are all girls).
The only women I am happy to be friendly with are colleagues (in a professional manner), family and ladies that my wife and I are friends with.
A lot of people baulk at this sort of arrangement/tolerance - but I bet it's quite common.