It's not about the expense, and calling it a pet peeve is very dismissive. It's about respect and personal space more than anything else.
Consider this: I'm more comfortable without pants on. When I work from home, in my own space, I'm usually not wearing pants. But when I come into the office, I put on pants, because I have some modicum of respect for the people I work with, and I know that pantslessness is going to make people uncomfortable.
Your behavior doesn't bother you, but ignoring the fact that it bothers others is the very definition of rudeness. Do you also lick strangers?