I think the main issue with his memo is the implication that diversity efforts were, at the time, problematic. The truth is, with such low diversity numbers even with the diversity programs, it's likely that any subpar candidates admitted probably equaled the subpar candidates admitted into Google at large. The numbers for gender diversity are even more skewed when viewed through the lens of the engineering department (which is the one relevant to the memo), which is unfortunately not available from the google diversity site like it used to be.
You can see the diversity breakdown for yourself here: https://diversity.google/annual-report/representation/
You can also see a longer analysis Damore's liberal assumptions here: https://www.wired.com/story/the-pernicious-science-of-james-...