The centuries-long existence of slavery, segregation (which was brutal oppression, including lynching), and racism isn't an "assumption", but indisputable fact. Occam's Razor is not a real arbiter of truth, but in this case it cuts the other way: Racism is the simpler and blazingly obvious explanation, backed by endless reearch and even the most casual observation. You really have to work to contrive explanations that don't include systemic and structural racism.
> poverty
Another outcome of those centuries.