http://en.wikipedia.org/wiki/Intersex
http://en.wikipedia.org/wiki/Klinefelter_syndrome
And, in fact, there's research showing connections between transgender identity (where socially constructed gender -- usually based on the more obvious indicia of biological sex -- doesn't line up with gender identity) and instances where the various indicia of biological sex aren't aligned in the usual way. Its not, as I understand, apparently the only factor in transgender identity, but it seems to be a significant correlate.
The idea that gender ought to be simple seems largely supported by the illusion that sex is simple.
This is a thing, there are people. Human sex is non-binary.