There is pretty conclusive evidence that taller men are on average more attractive than shorter men. However being more attractive does not necessarily mean you will have more kids. My theory is that people want to marry people of similar attractiveness, so if for very attractive people the pool of potential matches could actually be smaller, in part because attractive people are more choosy. Also in terms of evolution, if you have sex it doesn't matter how attractive your partner was.