For 350 years of US history africans and their descendants were enslaved. Native Americans were ripped from their land and relocated, often with genocidal levels of casualties.

After that, these two groups were substantially discriminated against in law, and other races were added to the mix to be given less rights than others.

Today, there are huge disparities between outcomes for different races in large part due to this historical discrimination. There's also an ingrained culture of stereotyping and discrimination that's hard to lift. It doesn't matter if you're the first generation of Americans descended from African immigrants who came in the 1980s... you still are impacted by this legacy.

The concept of affirmative action was to specifically counteract the effects of these negative, historical circumstances and provide a countervailing effect.

I can't speak to other countries, but in the US, it is definitely the case that poor people of color have a harder time getting ahead than equally poor white people. (I suspect it's similar elsewhere, but we are also a pretty racially diverse country, so the effect is larger)