Caveat: We don't actually sex-segregate prisons. We segregate based off of primary and secondary sex characteristics.
We segregate prisons by sex. Primary and secondary sex characteristics are observable characteristics that indicate one's sex.
Your argument is like saying "You don't buy a Mazda. You go to the Mazda dealership and buy a car that says Mazda on the back, interior, and on the manual. You don't actually know the car is a Mazda."