I think if this was just an isolated position or opinion it'd be easier to have some charity and understanding. That doesn't seem to be the case.
A good example of this is the international chess federation banning trans women from women's competition. [1] What advantage does higher testosterone offer for someone playing chess? That's where these concerns seem to be more "I just don't want to accommodate trans women" and less "I'm concerned about an unfair advantage".
[1] https://www.npr.org/2023/08/18/1194593562/chess-transgender-...