I also don't believe that appearance affects chances to pass on your genes. The ability to have sex does. But people with a lower than average appearance score (lets say), have the option of having sex with other people having the same problem. This also bypasses the fact that human beings are very complex animals, capable of falling in love with pretty much anything, including a piece of fiction and that when looking for a partner we also end up looking at other traits.
And please forgive me for my political incorrectness, but as an anecdotal experience, from what I've noticed the people that are breeding a lot (you know, the communities where families of 6 children or more are common place), well those people are downright ugly on average. Which makes sense, given that the people that are breeding a lot are usually the poor and they are ugly and in bad shape mostly due to not having access to health-care. Yet when it comes to having sex, they don't seem to care.