The main reason is that, if people of a particular group have been treated very badly by society for a long time, simply stopping that treatment is unambiguously a good thing, but it leaves that group in a disadvantaged position which can persist long after the persecution stops. Black people start off life (on average!) with a substantial disadvantage, so if we want people to have equal opportunities to achieve great things regardless of the colour of their skin, you have to do something to redress the balance.
This is why they sometimes advocate for preferential treatment for minorities: equal opportunity is no good if one group of people do not have equal access to those opportunities because they were born poor as a result of historical injustices.
An analogy sometimes used is that of a house on fire. If your house is on fire, you'd like the fire brigade to focus their attention on your house, rather than nearby houses that are not on fire. To an anti-racist, the idea of ignoring the colour of people's skin entirely is as misguided as the idea of fire-fighters focussing on all houses regardless of whether they are on fire.