We absolutely should not construct a credentialist edifice that says only people certified to have gone through brainwa---err, regent-approved programs--- can comment on a topic. That would prevent discussion on most topics, as virtually all topics of interest are complex and have many years of study behind them.
Overall, these comments are still criticizing the how instead of the what, which is what people do when they don't know how to criticize the what but want to express their offense anyway. It's much easier to criticize delivery and in fact it will always happen whenever anyone cares, because delivery is inherently contextual/subjective.
If Damore's paper was rejected from Nature or another peer-reviewed journal, that'd make sense, as it is not a rigorous academic work. It's just a conjecture on the state of diversity hiring and it expresses his reasoning for believing the way he does. If he is so wrong, it should be simple to disprove him, and we can all move on without anyone having to get fired.