You're speaking of "sex," not "gender" -- the former is biological and there are two of them; the latter is cultural or social and there could be arbitrarily many.
I say this while being a conservative; the semantic distinction is a useful one, although attempting to blur the distinction between a (biological) sex and a (cultural) gender never does any good.