Are people brilliant because they don't care or do they not care because they are brilliant? Seems to be a little of both...
There are things that don't matter (someone a bit clueless working earnestly) and things that do (scammers, violence, etc). The goal is to differentiate and respond appropriately.
IMHO, it's neither. If others want something from you then they pamper you to get what they seek, and the more indispensable you are the more they tolerate. Over time you feel no pressure to adapt your behaviour because those who deal with you feel compelled to adapt to you.