This is a huge thing that people seem to ignore when arguing "against diversity" (I put that in quotes because I don't think anyone actually argues against diversity -- they couch it in terms of meritocracy and egalitarianism, saying that the best people should succeed regardless of race and gender, it just so happens that the best people are overwhelmingly white and male.) I think Sam does a great job of presenting this as a thing of 'greed' -- or, more charitably, economic rationality.
In my mind, there are only two real possibilities (or, perhaps more realistically, two ends of a spectrum):
1. The VC ecosystem, as it stands, is almost completely unbiased and fair-minded with regards to gender and race. The massive disparities in terms of funding and founder backgrounds reflect that.
2. The VC ecosystem has some pretty systemic biases and prejudices (this is not to say that startup people or programmers or anyone is particularly evil; in fact, almost everything has systemic biases and prejudices), and as a result the current landscape does not reflect the best possible set of founders, startups, and opportunities.