People have different interests and it is totally fine. At our place some people are into Fortnite, some people into crypto, some people into playing the piano.
So when Fortnite people stay after hours to put some games in, it doesn't mean I have to stay back and play with them.
Most effective teams are usually ones with least communication impedance. The less you have to judge your words and go through diplomacy/"re-examination" dance in routine conversations, the more time you have to do your job. That's the reason "team fit/culture fit" is such an important criteria when hiring people.
Maybe you mean onus? And no, it isn't because that's exactly the sort of thinking that keeps the field as non-diverse and exclusionary as it is. What you're describing isn't some sort of property of an effective team. It's one of a hostile workplace.