If 1 in 4500 people are intersex by some definition, then that's about 1.5 million people alive today. Why should we force them to fit into a male or female box for the rest of society's convenience?
And even so, in (most) case of malformations, some malformation classes only affect male individuals and others female individuals.
But I agree that a good compromise would be, and only for this class of people born with a visible congenital ambiguity, to allow them to decide what sex they want to live as.
However, the sex of an individual who is born without any sort of ambiguity is useful data in a lot of cases, medical of course, and social to determine who has access to what segregated sex facility and so on and so forth.
I can see how this would appear reasonable at first glance, but it has so many thorny edge cases that it requires deep knowledge and systems-level thinking to address humanely.