When you come from the extreme tail of the bell curve that elite sportspeople come from, that small advantage becomes a large advantage. When you move the mean a little, the tails move a lot.
This doesn't just affect trans women but also intersex women. There's a definite tension in sport insofar as we expect elite sportspeople to be abnormal humans far from the mean. For males this is unproblematic: an abnormally strong male like Michael Phelps or Usain Bolt is still clearly a male.
But for females nature does not provide for us a clear line but rather a smooth gradient between female and male. The difference between an abnormally strong female and an abnormally weak male is not a line that nature always provides for us, it's a line that has to be drawn artificially and is therefore open to debate and challenge.