I would imagine there's a strong causal link between competence and being "dressed-to-the-nines"/"very sharp". Maybe signals of fitness and fitness here are to an extent the same thing.
When I was in grad school (physics) the female grad students used to talk about how you couldn't wear makeup or do your hair and be taken seriously.
https://en.wikipedia.org/wiki/Berkson%27s_paradox
In short: people can be correct when they notice an inverse correlation, because their world consists of people who got into the same university, or workplace, or club, as them. But the correlation is exactly the opposite in society as a whole.