That's a huge straw man there. The reality is closer to this:
One side: "We should acknowledge trans people and let them live their lives"
Other side: "Yeah, but it seems unfair to biological females to allow trans women to compete in competitive athletics"
First side: "You're denying the reality of trans people, and should be silenced."