I am from India and have lived in the U.S for over 2 decades. I have observed "affirmative action" arguments in two continents.

I am going to answer your specific questions,

>Why not just.. well, let poor people come first?

Racial justice issues are separate from economic justice issues.

Shouldn't this just be run-of-the-mill social democratic "lets hand out some extra opportunities/benefits to the poor" program?

Racism is a complex issue and in listening to Black/Asian/Brown Americans I have come to the realization that such programs have to be specific to each and every race because they experience racism in different ways.