The reality is "cultural fit" should never be used to make decisions. It is never right, either as a predictor of performance or ethically. Companies and interviewers already have an incredibly difficult time figuring out ways in which to judge qualifications, even when standard tests and procedures are available. To add in this nonsensical idea of "cultural fit," something that can't be measured, quantified, qualified, or identified, and to make that a selection criteria for applicants not only shirks the duty to hire the most qualified, but gives the company and interviewers an unchallengeable way to reject applicants, an option that otherwise would--and should--be illegal.
tldr: Cultural fit is discrimination, it cannot predict job performance or any otherwise meaningful information related to a candidate's skills and abilities, and should be illegal as its only use is to deny qualified applicants jobs they qualify for.