Everywhere I've worked and to my knowledge everywhere I've interviewed has considered culture fit. I don't think it's always as pernicious as you're describing. When done right, hiring for culture fit means some combination of "don't hire people who aren't excited to work here" and "don't hire assholes, no matter how well they code." Both of those are important filters, and I don't think either of them is unfair to the applicant.

Unfortunately I've also seen teams where it meant "don't hire anyone who isn't a workaholic," and teams where it meant something like "don't hire anyone who is going to wreck the hard-partying vibe of the workplace." (I've never seen anything nearly as crazy as "don't hire Yankees fans.") It sucks to get rejected for a job because the current team all hangs out and they don't think you'd dig the vibe. It would also probably suck to get hired into a team that's also a close-knit social group that you aren't particularly interested in joining. I think the takeaway is that if you get rejected for a team for being a bad cultural fit you've probably dodged a bullet anyway. Go find a job with a team that knows how to make people of all ages and stripes feel welcome.

EDIT: I just remembered my favorite "culture fit" interview question of all time. I was head of engineering at a startup that was interviewing around with potential acquirers. I was talking to a senior engineer at one of the companies. He ran me through a couple simple warm-up questions and then hit me with the big one: "If we acquire you, are you likely to up and quit in two months?" I stuttered for about two minutes.

It was a great question! In hindsight, I almost certainly would have. We both dodged a bullet there!