I agree with you. If someone (male or female) were writing about an interaction with the genders reversed, you probably wouldn't hear anything about how good-looking the guy was or what he was wearing.
The (unconscious) sexism is not in finding her attractive (obviously) but in the fact that her attractiveness is treated as a relevant detail. How would it be different if she were unattractive? It wouldn't.