Why is it annoying? Who told you that you need to feel bad?
Because it affects his concentration? He already expressed that in his post
" or at least why can't it be nothing at all so it doesn't interrupt my thinking"
Imagine if everyone around you in your workplace would think about how you're dressed instead of what you're actually trying to talk to them about.