1. I think it's important also to recognize that the "science" he cited was not fool-proof. In the way that most science is not (for example, check out the 50 year long & still on-going dispute with Phillip Morris about the harmful effects of tobacco). And that moreover, the specific reports he cites can certainly be disputed on the methodology (where are minority women? where are women of different socio economic classes? where are women working in technical fields whose performance do not correspond to standardized test scores? etc)
2. Even if Summers was absolutely right about sex difference between men and women, some of the outrage directed towards him came not from particular concern with his science, but the effects that such statements would have on the construction of a liberal democratic society (which Harvard, despite it's many follies, is mostly invested in). A society which operates on the truth that women just can't do math as well is one which condones expectations that adversely effect female participation in communities of scientists, mathematicians, engineers, and tech-related fields.
*edit for formatting