It can be a cover for clear racism/sexism/agesim etc. but can also be used to discriminate on less clear terms - i.e. maybe all the co-founders at a startup went to Yale, Harvard, MIT etc. but the candidate is a self taught programmer who comes from a poor background in the Midwest. If the candidate shows that he is just like the co-founders (i.e. has the same interests, outlook on life etc.) then he will be a culture fit. But if the candidate has four kids at the age of 29, goes to church every sunday, hunting on the weekends etc. he will not be a culture fit.
In essence, "culture fit" is the result of a company hiring candidates based on aspects of them that should not even be considered at all. It shouldn't matter if the candidate has a large family or no family at all, if they candidate is religious or not, what the candidate political beliefs are - these are all things that should really be kept out of the workplace.