The reason we have women's sports in the first place is essentially a different form of "you have to be below 2m" in order to promote diversity in sports, so disallowing trans people doesn't seem like a much bigger step than disallowing 50% of the population.

Personally I don't have a strong opinion on this, especially since trans women taking over women's sports is not a problem yet anyway.