He does a provide a source in the form of a wikipedia article that uses actual citations [1]. The version with sources (gizmodo removed them in their first publication of the memo), is located at [2].
As for your comment on his "blatant disregard for social context", he explicitly mentions that he believes that biases should be corrected and that society probably also is to blame [3], he just doesn't believe that biological factors does not play any role at all.
[1] https://en.wikipedia.org/wiki/Sex_differences_in_psychology#...