I think "fit" is a cop-out when left unspecified. It's a reasonable way to reject someone when you can clarify why someone is unfit. I think rejecting a hire because you don't like them isn't okay unless you understand why you don't like them.
Understanding that someone would be bad because they're overly argumentative, can't take criticism in a calm or level-headed manner, or because they're prone to interpersonal conflict is a legitimate way to reach the conclusion that they would be a bad fit. But if you get the cabal together and accept a simple "I don't like them" without requiring details, it becomes ripe for abuse; if only by your own subconscious judgements.