If this was your takeaway, then I did not explain it well enough. Every individual has a distribution of ability. Some are better at math, some are better at sports, some are better at reading, etc. 2/3rds of boys are better at math than they are at reading. 2/3rds of girls are better at reading than they are at math. However, girls are actually better than boys at both reading and math - it's just that they score better at an even bigger margin at reading.
Is it inconceivable to think that the fact that girls are better at reading than math 2/3rds of the time makes girls more likely to prefer reading as compared to math (and vice versa for boys)? That's the point that Damore was making: the distribution in ability affects boys' and girls' preferences. It does not say that the average girl has less ability than the average boy.
If you want to split hairs, you could say that making this argument is sexist because it says girls score better than the boys on average in both reading and in math. But I get the sense that this isn't the angle you're making.
> Extraversion expressed as gregariousness rather than assertiveness. Also, higher agreeableness. This leads to women generally having a harder time negotiating salary, asking for raises, speaking up, and leading. Note that these are just average differences and there’s overlap between men and women, but this is seen solely as a women’s issue.
> - Neuroticism (higher anxiety, lower stress tolerance).This may contribute to the higher levels of anxiety women report on Googlegeist and to the lower number of women in high stress jobs.
Both of these points have to do with specific parts of the job (salary negotiation, asking for promotion, dealing with stress). These factors exist in all jobs, not just tech. Furthermore he later offers suggestions to try and mitigate this - he brought these things up because he wanted to offer positive changes.
The point remains: Damore did not write that women "aren't physically suited" (~~your words~~) to tech work.