Sex is a binary because that's how anisogamic sexual reproduction works: females embodying the reproductive role organised around large gamete production, and for males, small gametes. This still accounts for disorders of sexual development too: a male with underdeveloped testes is still male, and a female with non-functional ovaries is still female. In some extremely rare occurrences of chimerism, a person could reasonably be said to embody both.
But that's not what anyone means when they use the term "non-binary" as an identity - it's expressed by people who are unambiguously male or female, but who feel that they don't fit into the cultural gender stereotypes of masculinity or femininity.