It's not a problem for anyone other than the overly self-conscious, who are looking for reasons to explain their anxiety. Literally no one cares.
Another boss I had any a different job told me he fired someone for not partying with everyone else. If you went to these parties and didn't drink, the boss would harass you all night.
I agree with you that many people are just self conscious, but it's not exclusively that way.
The tl;dr is that people who are insecure will attempt to rationalize their feelings, and this frequently includes feelings of judgement and persecution from others where none exists.