Unless you can quantify it, it doesn't really matter whether the criteria is "would enjoy drinking with" or some other subjective X-factor.
The big problem with "culture fit" in interviews is that it gives you an escape hatch to avoid the conclusions that concrete objective assessments provide.
Candidate did poorly technical qualification challenges, but man were they great to talk to at the interview! Fuck it let's hire! Candidate did great on the technical qualification but hrm they have a family and live out in the suburbs? Meh let's pass this time and maybe come back to them later.
You might as well not be hiring rigorously or objectively at all.
My experience running concrete work-sample-based resume hiring programs (we're hiring now!) is that every force in the institution you're working for is going to push you away from trusting your assessment results. And if you can't rely on those results and learn from them, you can't iterate and get better at hiring. And indeed, most organizations never do get better at hiring, and draw from the exact same pool of "conventionally hirable" candidates everyone else does, and it's a disaster.