Nothing in biology is really binary anyways. Male and female are just useful groupings because we’re a sexually dimorphic species, but as with anything in nature, that’s mostly a spectrum anyways, with strong representations at the anchoring ends.
Sexual dimorphism in gonochoric species is a different type of thing to this.
Its actually rather astounding how much binary pairs play a role from the very core of physics.
On the other hand, the XX vs XY karotype (chromosome set) is just very strongly correlated with sex.
The master switch for sex determination in humans (and most mammals) is the SRY gene, usually found on the Y chromosome. Its presence or absence determines whether a developing embryo takes the path towards producing sperm or producing eggs.
SRY can migrate to the X chromosome (resulting in males with an XX karotype) or can be broken by mutation (resulting in females with a XY karotype, usually infertile because other stages of development of the reproductive system depend on having two X chromosomes).