Your two issues are issues because, even without any prejudice, there are physical differences that we have to handle and it's not easy to figure out how to be fair to all parties.
In the case we're talking about, for example, the trans women fractured a woman's skull. Broke her skull. Was the skull the problem here? Should it have been less breakable to match the rhetoric?