I'm pretty sure it's the other way around. If as a white person you punch a black person because you don't like the car the person is driving, that's not racism, that's just some weird aggression. It's not based on the concept of race. But if you punch them because they're black, that's obviously racism.
Same thing here. If you're taking an action that indirectly causes harm to mostly non-whites (more specifically to poor people, some of whom are white) without the intent of causing harm, that's not racist. It's only racist if you feel that the group that's being harmed is racially/ethnically inferior to your group and that that justifies your actions.
That isn't to say that knowingly causing harm to people, even without any racism involved, is OK. It's not. But it's the intent that defines racism.