It's not illegal, except if you discriminate against legally protected classes of people. For example, in the U.S., it would be illegal to say that women or blacks or Muslims or people over 40 "don't fit in to our corporate culture" because sex, race, religion and age (over 40) are among the attributes covered by non-discrimination laws. However, it's perfectly legal to not hire anyone who has blue hair or who doesn't wear a suit. Not necessarily a great idea, but it doesn't violate any laws.

"It's just assumed to be a good policy."

That's certainly not a universal assumption. Many people on HN have maintained that the ability to do a job well is much more important than "cultural fit" (and I'd agree with them). And many have observed that if companies hired solely for the ability to do a job, it would be much easier to find qualified candidates.

"it's like being on a date"

No, working with someone is not like being on a date. I'm happy to work with lots of people who I'd never go on a date with. I really don't care what my co-workers' preferences in food, music, movies or sex are.