And someone who doesn't do any work might not "fit in" with a team of people who are doing work, but I'm not going to hire them for the sake of !current_culture.
There's a reason it's a "culture fit" or "match" and not "culturally identical" or "culturally homogeneous." The phrase as it is is sufficient and can describe a healthy way of hiring people who are compatible with your current team. If we assume our current team has a healthy culture, finding people who are compatible with it, that can effectively perform their professional duties without disrupting the ability of other employees to do the same (and, ideally, improving the ability of others to do the same), is exactly what we're looking for when hiring someone (other than technical aptitude).
Sure, it can also be used to illegally or unethically discriminate, but so could any concept you could conceive of to describe "they won't perform well with our current team." Before "not a culture fit" was a popular it was "not a team player."