Part of what you signal with your wardrobe isn’t just that you care for yourself. You’re signaling to others that you care about how you appear to them. We can’t expect other people to ignore that signal because showing that you care about how other people see you is a proxy signal for caring about other people.