"I don't see color" doesn't feel helpful to many people of color. If you believe that there are prejudices, and you want to fix them, saying "everyone immediately needs to stop being prejudice and everything will be better" isn't actually a solution we can enact. What if people don't suddenly decide it's in their best interest to not be racist?
*Edited to remove off topic comment
You're right that we've made significant progress on racial issues. That doesn't mean more can't or shouldn't be done. Furthermore it's not clear from your slippery slope argument that the disadvantages of reverse discrimination (of which there are of course some) are actually more significant than the advantages.
And yet, in this country we did just that for hundreds of years. You don't just wave your hand and erase that legacy. We are in a transitional period that will hopefully help ease us into a society where these measures will truly have no place. We are not there yet.