The correct reading is that there is biological difference both between the sexes and inside each of the sexes.
This is a reference to the identity politics you see especially in 2ndwave/3rd wave feminism which claims that almost everything is a social construct.
He says there are IQ differences between people. Who are these "people" if not men and women? He either means men are smarter or he just randomly went on a tangent.
Let me ask you, when he says there are IQ differences between people, which groups of people do you think he means?
He is referencing the people who claim there is no biological differences and that almost everything is social constructs.
If you want to understand what he is talking about here is an example
https://www.nytimes.com/roomfordebate/2015/06/16/how-fluid-i...
For someone to say that people who are already in a career, that is, already selected for their ability, are less suited because of their sex is kind of absurd.
That tidbit is a paranthetical in a sentence about the psychology of left vs right. His thrust is an illustration of how ideology and science are conflated.