Dress is form of self expression - intrinsicly so.
Rationalism is absurd, and this is a great example of why.
Humans almost never do anything on a purely rational basis. Everything we do is directed by subjectivity, emotion, aesthetics etc. - it's better to be honest about this and embrace it, rather than fooling one's self that one's aesthetic preferences are anything more than subjective.