Others have expressed thoughts I had as well, but to add:
I used to basically have your opinion, and then had two experiences that shifted my perspective.
The first was a colleague who was known for being overly frank, to put it mildly; for not having any internal controls on sharing what was on his mind. He was (and is) a brilliant individual who is every bit as competent as people much more accomplished than him. I watched him lose out on job prospect after prospect, and initially sort of thought "well, being able to work with someone easily is important." Eventually, though, I realized that what is far more insidious are all the narcissistic psychopaths who are socially skilled. This guy, while brash, actually was well-meaning and had a heart. I'd far rather work with him than many of the other horrible individuals I've worked with; he's at least as deserving of better employment, if not more.
The second was a case where we were hiring, and this brilliant individual applied. Everyone agreed he was heads above the best -- this being a faculty position at a research institution. Eventually everyone started complaining about how socially awkward he was, how his presentational style was difficult, and how he did these weird things in interviews. The thing was, he wasn't mean or cruel or anything like that, or even markedly eccentric, just a little weird. Someone objected to this, pointing out that it was bizarre for us to be dismissing someone who was described as a genius in letters, and that we agreed was such, because of socially unusual (but not difficult) behavior. The question this person on the hiring committee raised was "what are we actually wanting? What is our mission?" The applicant didn't get the job, and it always struck me as unfortunate.
I guess my point is that even if someone is unpleasant to work with, sometimes that's not the most important thing, and even when it is important, the way that they are unpleasant matters. This isn't even getting into how capricious and cruel people can be in their assessments of others' behaviors.