What you said about the negative stereotypes about blacks is unfortunately very true. But: affirmative action does not fight against the stereotypes, it fights against the underrepresentation. As some other comments said, it may even reinforce the stereotypes (I am not sure if it works that way, however). While we cannot completely avoid stereotypes, we can try and make sure they do not affect how the people are actually treated. Well, to some extent, hopefully large extent.
You mentioned no stigma on a white child. Some people do talk about rednecks and so on. And indeed there are programs similar to Affirmative Action intended for, say, people from underprivileged backgrounds in the mountains of Kentucky. Another example is programs for Native Americans (or "first people", or whatever today's PC name is). They have been traditionally underprivileged, but I do not have they have a stigma, not for a long time.
accept that you will not get into those schools
because chinese immigrants and jewish people
will probably get the best scores