If you have very unattractive qualities and because of that people are not selecting you, they are not punishing you. If someone doesn’t want to be with you because you admit you’ve cheated in the past but don’t plan to anymore, that is fair. If you carry genes that make you more likely to seek out other partners and they don’t want that, that is also fair.
Your genes that set physical attractiveness have little to do with just how attractive you actually are, who else may or may not find you attractive, and your chances for successfully passing on your genes.
Eg, your genes aren't the entire story, maybe not even most of the story.