We need to work together to make culture-fit a bad word. "Culture fit" means discrimination along dimensions that shall not be named, for were they to be named they would be illegal.
Short of always requiring a detailed reason why a candidate was passed over, there will always be the possibility to fudge an illegal act of discrimination to look like a legal one (and even a detailed explanation requirement wouldn't fully fix the problem).
You want to work with them because they are competent and results-oriented? Good.
You want to work with them because they are eager and capable? Good.
You want to work with them because they look kind of like you and like the same beers you do? Nope.