Then it should be based on poverty, not race. The people that didn’t deserve to get in to those top tier schools can make just as much money after graduating from a “lesser” school.
You'll find a lot of us out there who are vehemently opposed to race-based affirmative action but would enthusiastically support wealth/class-based affirmative action.