And the key is to consistently not be an asshole, otherwise those things will accumulate and you will eventually get rejected. (I'm not accusing you of being an asshole.) If you're nice 90% of the time, but lash out or say shitty things 10% of the time, that's more than enough to get eventually rejected.
My son, unfortunately, is like this. 95% a sweet kid but 5% really, really shitty and saying mean things. We are working on it. He started out immensely popular but over the course of this school year, his classmates look at him lukewarm now, instead of being his close friends, and it's entirely his issue.