There's a certain irony where, although it sometimes feels lonely being a floater, such people often seem to have higher status when viewed from the outside. They're confident enough to not be defined by their peer group, after all. Few people are, and many people wish they were.
On the one hand, one tacitly feels contemptuous towards the other group and tries to come across as dispassionate in their appraising comment. On the other hand, one feels considerate (or neutral) and simply calls a spade a spade in their appraising comment.
When you take the entire context of people's interactions over time into consideration, the former is likely to be hurting (overall) and the later is likely to be helping (overall) even though certain specific instances may come across as the opposite.
tl;dr - feelings and vibes (over time) matter more than words (in individual instances).
<knockout_punch> ::= <personal_attack> <you're_wrong>