These people who are at the bottom will stay even more firmly entrenched at the bottom, will commit more crime and continue to justify the stereotypes.
If you believe that blacks are dumber than anyone else (like the linked article), then you have to support affirmative action, otherwise you are setting up a society segregated by intelligence, which will also directly correspond to skin color.
So blacks will always be at the bottom, and people of black skin will automatically fall into that category with very little chance of getting out, even if they are clever, as there will be many people justifying the opposite stereotype.
Imagine all colleges had 5% less space for students. That's the only effect of affirmative action. If you want to campaign for something, then campaign that 5% more students should be taken in - one for every student who got in as a result of affirmative action. That's a win-win.