> No one is worried in the slightest about letting trans men compete in men's sport.
By and large for most disciplines, there's no separate men's leagues or men's competitions. So women and trans-men and everyone else is mostly already allowed to compete in men's sports.
See eg https://en.wikipedia.org/wiki/Judit_Polg%C3%A1r
> Traditionally, chess had been a male-dominated activity, and women were often seen as weaker players, thus advancing the idea of a Women's World Champion.[16] However, from the beginning, László was against the idea that his daughters had to participate in female-only events. "Women are able to achieve results similar, in fields of intellectual activities, to that of men," he wrote. "Chess is a form of intellectual activity, so this applies to chess. Accordingly, we reject any kind of discrimination in this respect."[17] This put the Polgárs in conflict with the Hungarian Chess Federation of the day, whose policy was for women to play in women-only tournaments. Polgár's older sister, Susan, first fought the bureaucracy by playing in men's tournaments and refusing to play in women's tournaments. In 1985, when she was a 15-year-old International Master, Susan said that it was due to this conflict that she had not been awarded the Grandmaster title despite having made the norm eleven times.[18]
> Or letting trans men into men's toilets or spaces ...
That one is interesting, because women's toilets are usually all cubicles, and men's toilets is the only place where you could actually see anything.. Btw, it's not just trans-men that get a free pass here, but cis-women also often go to men's toilets when the queue for the women's toilets is too long, and nobody bats an eyelid.