I feel like that's pretty likely a bigger element of it when I was younger, though I'm sure it always figures in to some degree. Although part of it is circular: when people all treat you like you're nicer/smarter/etc. you feel more confident, too.
But some of it was just a bit too much of a stretch to just be confidence. I'm /generally/ a pretty confident person, and comfortable with people, and the difference in how many people suddenly can vs. can't find the time of day for you is pretty dramatic.
And, of course, some of it was just not subtle at all. I'd been told pretty frankly I was too fat to be a doctor. Because, you know, clearly my personal struggles with my weight have a direct correlation to my ability to diagnose your ailment. (Okay, I'm still salty about that one.)