The thing is that if someone goes to Harvard, their children are more likely to succeed. It's effectively an inheritable benefit; it's just not actually just that "people X did Y they deserve Z"; but what do you want to set up for people who will be born 5 or 10 years from today; do you want it to be the case that you can statistically predict the success of one of those future-children will just be less, and the reason why can be directly tracked back to historical overt racism?

Obviously tons of people are uncomfortable with affirmative action (which isn't as severe as your reductive description makes it); if you have literally any other way to reduce the measurable racial-based disadvantages that exist in our society there's a ton of people who would be very eager to hear it.