It's important to understand
why race of all things is so foundational to systemic injustive in the US.
The original sin of the US was the Atlantic slave trade. It so happens that almost of the people swept up into that slave trade were dark skinned. And, because that slave trade was so monumentally huge in terms of how many people it brought to the US, a very large fraction of dark skinned people in the US today and all throughout history either were slaves or are primarily descended from them.
In order to morally justify the enslavement of millions of people, white slave-owners manufactured a justification that people of certain races are physically, mentally, and morally different from whites. They created a (factually false) narrative that skin tone didn't just mean skin tone but also implied all sorts of other things about the person.
That narrative is profoundly embedded in all aspects of US politics, culture, and institutions. It's been there since before the three-fifths compromise which is directly in the US Constitution. It's everywhere.
So it's not just about skin tone. It's about generations of Americans growing up in a culture that says people with a certain skin tone are less human, and then building social institutions predicated on that as fact.
Culture and institutions change very slowly (especially when many participating in them actively benefit from their current state) and it takes a lot of work to root something like this out.