Instead of being annoyed, perhaps you can find joy in watching what your mind perceives as beauty, just like people look at cars or sunsets?
My grandmother always used to live up when she saw us and complimented us on our looks, although I'm sure they were nothing spectacular. After having children myself, I'm beginning to understand her - there's something satisfying watching other people. I think we should accept those joys we receive for free here in life.
-- Friedrich Nietzsche
Imagine if everyone around you in your workplace would think about how you're dressed instead of what you're actually trying to talk to them about.
Because it affects his concentration? He already expressed that in his post
" or at least why can't it be nothing at all so it doesn't interrupt my thinking"
I can't even conclusively tell the difference any more.
A) Noticing a woman's beauty when you first encounter her.
B) Making a 'valuation' of a woman 'solely' based on her looks.
What's really deeply disturbing about the above is the mind-control aspect, driven by shame. Not only must you censor your words, but you must also hate yourself for your involuntary thoughts. This is total ideological possession, as Peterson puts it.
And it's so, so familiar. I swear, soon enough progressive men will be wearing chastity belts to help banish impure thoughts, or hair shirts to make up for their privilege. I am not even sure I'm joking.
Personally I don't think that is a particularly good way to live though. The mind is our only refuge from the rest of society, and to attempt to censor your mind so it's more acceptable to society seems like a really sick thing to me.
To me it doesn't seem that OP is arguing against self-regulation or self-improvement in general, just self-regulation against conforming to society at large. That isn't to say one's own personal values aren't influenced by the values of society.
Thinking about it more the whole issue is incredibly complicated.
EDIT: Just to add that your comment's two halves are not logically connected. Even if we do act in this way, it doesn't make it a "critical part of what we are", or if it does, there is no reason that "critical parts" cannot be altered if need be. In the early 19th century most people would have thought that a 10 hour working day was too liberal - an 8 hour working day would have been unthinkable; but that "critical part" of people's way of perceiving society was not set in stone, as it turned out.
Whether this all derives from biology is hard to say - however the alternative is to suggest that we are all somehow indoctrinated to value physical attractiveness and would otherwise be saintlike in ignoring things that suggest genetic unfitness, such as obesity, small stature, disfigurement etc. Look at the animal kingdom - I don't think animals are gauging partners on their ability to tell a funny joke, or having the correct politics. The sex drive is one of our most basic drives. To think it's not still driven by basic prerogatives does not seem reasonable to me.