> Those people are the exception
If there are exceptions, then it's not a fact. At best, it's an approximation that works well in most situations, and is subject to exceptions where it is simply false.
[1] the prevalence of intersex is seen to be about 0.018%: https://www.tandfonline.com/doi/abs/10.1080/0022449020955213...
This is a parochial viewpoint.
Are you aware that other species have a wide variety of mechanisms for determining sex? Which means that evolution can and does result in the modification of fundamentals, at least if you take common descent for granted.
This can only happen if a mostly bimodal distribution is not entirely bimodal - what would provide the material for evolution to something different otherwise?
Hence, the outliers are not insignificant at all, except according to some arbitrary value system.
I don't have a reference, but I vaguely recall reading that the asymmetry between the X and Y chromosome has developed under evolutionary pressure and eventually the Y chromosome may disappear entirely.