Affirmative action will prefer a middle-income black person to a low-income white person. It 'evens the score' for some poor people, but not for others.
There are plenty of systems that benefit some areas of society and not others. It sounds from your comment that you're all for an even distribution of rewards across society. Is that a fair characterization? Why is this case particularly egregious to you compared to other inequalities?