I'd rather live in a society that made it an imperative to have more students from more ethnicities be eligible for the best universities every year.
That requires addressing inequality earlier, and more thoroughly, and across more axes, than an approach that is, in the final analysis, about setting quotas based on ethnicity.
Affirmative action can do everything it was designed to do while having essentially zero impact on inequality in society at large - it's always seemed like a cop out to me.