He did cite evidence for his claims, but the most widely circulated copies of the memo removed his citations.
Do you know of any?
>And that is what the Big Five represents: a consistent model of how humans reflect individuality using language, no more. There were no considerations of findings in neuroanatomy, neurochemistry, experimental psychology, observations of behavior of people or animals in real situations – none of this was used at the research stage leading to the development of the Big Five. In this sense we can say that the Big Five does not represent the structure of temperament or the structure of biologically based traits, even though lexical perception reflects some elements of it.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3903487/
The big five can still be used for self-reported correlational analysis, but Damore used it in an argument against social constructs, when the method itself can be heavily biased by the social constructs he's arguing against.
Though it is probably true that the lawyers in the case didn't read the citations. It wasn't their job to determine whether Damore's statements were scientifically accurate, and they stated that his statements were discriminatory and constituted sexual harassment, regardless of the scientific references and analysis.