I agree that in principle anyone can do what they want, and I hope that in our long term our society can actually realize this ideal, but in practice an individual's decisions are informed by the mores and attitudes of the culture they find themselves in. So the conflating factor is our society's history of discouraging women from pursuing technical paths, the (older) insistence that they be homemakers and take on more traditionally feminine roles (like K-12 teachers). Yes, we no longer overtly say such things, but the effects of these attitudes persist across large subpopulations and ethnic groups. It seems very likely to me that traditional gender roles and attitudes about what jobs a women is capable of doing has caused fewer women to pursue math/physics (e.g., see the statistics here:
https://math.mit.edu/wim/2019/03/10/national-mathematics-sur...).
Also consider the very recent history of breakthrough results coming from female mathematicians like Lisa Piccirillo and Urmila Mahadev. How much human potential are we leaving on the table, untapped?
I see no "easy fix" for improving representation in higher education other than affirmative action. It's one of the few knobs that we have to turn without much more significant changes to society like implementing UBI or making the average work week much shorter. While it may hurt people like me who happen to be well represented, and prevent me from joining the institution "I deserve" to be in, well maybe that's better for society in the long run.