The debate is really around how the handling of intersex and transgender athletes intersects with the original purpose of creating a separate category for women.
This is exactly my point. Men with unusual characteristics are celebrated, but women with unusual characteristics are excluded into a non-competitive category.
You can justify it if you'd like, but in a practical sense, no man will ever get to the Olympics only to be turned away because they don't genetically qualify for competition. This is an indignity reserved only for women.
No adults are training their way to the kid-lympics and then getting cut open and surprised by the count of the rings.
Also, the idea of "fairness" is overstated, a naturalist just-so fallacy. Is it "fair" that some male athletes are taller or shorter than others, or have other genetic advantages, for example?