The concept of affirmative action has never been about individuals. It's a RACE-based correction for RACE-based discrimination. I don't know what makes you think it's not racial.
Your solution for the societal crisis you face is to keep those at the bottom at the bottom, just so that your kids will have a 0.002% higher chance of entering one of a few colleges.
You have a large underclass of people whose parents were raised in institutionalised dicrimination. These people are the lowest caste of your society. There is a policy in place that favours a very few of these people to get into spots they would not otherwise get into.
Without this, the status-quo would continue to exist. The problem is this - the status quo is pretty bad. Blacks are less educated, less likely to get jobs, need more welfare, have higher crime rate.
To me, getting more into college seems like it would solve some of those problems.