If you recognize people who are good at their jobs but who are passive in other ways and you treat those people well while others disparage them for being passive, If you can handle the social bullshit for those people and let them do their jobs, those people will feel real loyalty towards you, and not just because they feel good about you; if you can separate the 'do your job' from the 'social bullshit' those who can do the job but can't or won't deal with the social bullshit really will be much more valuable than they would have been on a team where that social bullshit was expected of everyone.
If you value aggressive people over passive people in the general case, you are leaving a lot of value on the table.