> I only interact with colleagues and I can't imagine them 
    > caring what I wear.

We do. We don't tell you, and if we're any good we don't show it, but we notice and we draw conclusions about you.