The "gate" you're talking about uncontroversially belongs to each individual in the form of their bodily autonomy. Yet women are uniquely cast as "keeping out" certain people. That's not a matter of dimorphism: that's a matter of mens' bodily autonomy being assumed, and womens' being up for debate.
In other words: for the sake of argument, sure, there's a gate. But only women are viewed as gatekeepers, as though keeping others out undermines equality, rather than being part-and-parcel of having the equal autonomy to choose one's partners.