You'd be surprised at the number of straight-acting gay men that you interact with everyday.
You're basing your judgments on the flamboyant feminine gay men, the ones who you know are definitely gay. That's a very biased sampling to base your judgments on.