You ask "Where did he say that?" and I'd be surprised if there's any one succinct place -- it's one of the two main topics of the whole memo, and the memo does not have a coherent topic sentence or even really a coherent argument, so I think it's likely absent.
But, like, Damore makes a case to the effect that "biological explanations can't be ruled out" and then reverses those weasel-words by suggesting that his biological explanations be used to guide policy by, say, encouraging pair programming which he supposes to be something that women are likely better at on average. This sort of move suggests that he thinks the biological effects that he's citing (see note [1]) are big enough to guide policy, which they're not. You don't need to take my word for it -- the main author of the article Damore is citing was asked to read Damore's memo and this is his take on it [2].
Of course the problem is even worse in that this article which Damore used to write his article is psychological; it is based on doing a personality test in a bunch of different nations. That makes it very hard to conclude anything biological about it, so every time that Damore mentions "biology" in his memo, that is an interpretation of his own devising. The original personality-study article also interprets its findings biologically but it is really tenuous [3]. In fact neuroscientists have also been studying the brain and they have not found a clear biological difference between male and female brains [4].
[1] He gives a summary of a Wikipedia summary of an article by Schmitt et al. (2008). The PDF is freely available by the university at http://www.bradley.edu/dotAsset/165918.pdf but the sample sizes were I believe later corrected as an erratum, so I am not sure which one this has.
[2] Schmitt, evaluating later research as well, summarizes by saying that sex differences are only "accounting for less than 10% of the variance" and that using this to guide policy is "like operating with an axe. Not precise enough to do much good, probably will cause a lot of harm," in an article at http://quillette.com/2017/08/07/google-memo-four-scientists-... .
[3] The argument in the article involves their surprise that the majority of the discovered effect apparently disappeared in Africa and East Asia. Their interpretation is literally that those cultures are so much less economically developed than we in the West are, that their women must feel so much less free to just be themselves, and therefore they act more like men as a sort of baseline survival tactic. Read the paper; it's a very 'you cannot possibly be saying what I think you're saying, can you?' type of experience.
[4] See the links in the article https://www.newscientist.com/article/dn28584-a-welcome-blow-... for a nice summary.