I think people will complain about fairness even when they're not affected, because the tech community tends to have a strong knee-jerk reaction to this sort of thing. But it would be foolish to not consider diversity as one of many metrics to gauge how effectively one is avoiding costly biases.
The entire campaign is centered around encouraging women to apply more BECAUSE they can be good founders, not because they are women, and men feel bad for not including them.
EDIT: ahh sama beat me to it.
Competence knows no race, color, creed, gender, or nationality. It may be found in anyone. I'm greedy too, and want to receive the benefit of competent people of all kinds who are allowed to flourish and succeed. Here's Sam:
> We want to fund more women because we are greedy in the good way--we want to fund the most successful startups, and many of those are going to be founded by women.
and
> Again, we don't do this for the sake of diversity. We do it because we want to get the best people, whatever they're like.
I'm probably not the only one who had to look this up :)