Are you suggesting that there's no phenotypical relationship between appearance and psychology? I mean hormones effect both fairly directly. The notion that we can't identify patterns based on outwardly visible aspects of people is a factual claim propped up on MORAL grounds, and not empirical validation.

The idea that people who "hit the gym" might differ from those who don't psychologically, precisely because of either the physiological results of working out, or the resulting social standing that results from being judged as attractive or dominant, scares some people. And to engage my own personal pattern recognition, the people it scares the most are those who are judged as not physically appealing or imposing, and perhaps would benefit from "hitting the gym."