Because that assumes "poor people" get "help" in a uniformly fair and anti-racist way, and that's never really true in the US today.
If you help "all people" with "no regard" to race, you have just participated in favoring white folks over all others, even though you likely don't realize it. The systems by which you choose to "help" all have various types of racism built-in, and you will have racist results as output, even if you yourself never directly try to commit such an act. (This is what systemic racism is, sometimes called 'institutionalized racism' https://en.wikipedia.org/wiki/Institutional_racism )
Affirmative Action, while not perfect, is one of the few things ever tried that actually accounts for this. It's saying, "you can't be more racist than X" where X is some kind of objective metric (say, "percent of enrollment by race"), and it does not care which of the thousands of people or systems involved are causing the issue, it attempts to force-corrects for it.
It is an emergency stop-gap, until such a future as that result is already happening naturally, making it redundant. The fact that we still depend on it ~60+ yrs later, is sort of living proof that we haven't really dealt with systemic racism yet. (As if all the other evidence, between housing, employment, police brutality and murders, etc, wasn't already enough)