If you have very unattractive qualities and because of that people are not selecting you, they are not punishing you. If someone doesn’t want to be with you because you admit you’ve cheated in the past but don’t plan to anymore, that is fair. If you carry genes that make you more likely to seek out other partners and they don’t want that, that is also fair.
Your genes that set physical attractiveness have little to do with just how attractive you actually are, who else may or may not find you attractive, and your chances for successfully passing on your genes.
Eg, your genes aren't the entire story, maybe not even most of the story.
Genetic expression is influenced by many things, some that we understand and some that we don't (yet), but it is highly plausible that this can be (and already is) influenced and optimised throughout people's lives.
People's negative reaction to your proposal is obviously because people think it's dystopian to think that the genetic hand you're dealt at birth is highly restrictive and deterministic in terms of your potential life outcomes.
Policies like the one you describe would lead to a horrible society, because anybody who comes to believe that they are doomed to a miserable, futile life will not contribute positively to society and may well contribute very negatively. I suspect much of what we're seeing playing out around the world right now is at least partly a result of this phenomenon.
If it turns out that people's genetic expression is far more important than their genetic blueprint, and that their genetic expression can be influenced and optimised to lead to improved life outcomes, then people will be motivated to contribute positively to society and society will be vastly improved.