One thing that helps with this: getting old. You just stop worrying about what other people think of you. All the drama and gossip and cliquish behavior just gets so boring.
Why do you think old fat guys walk around naked in the locker room at the gym? They've certainly got nothing to show off, but they don't give a shit.