Segregation would still probably exist without the feds stepping in therefore let's spend trillions of dollars a year driving our economy into the dirt.
Then on the other side, you have the Dred Scott Decision, the Fugitive Slave Act and many other Federal moves that prolonged slavery in this country. Would it not have been for the northern states and churches, the Federal Government would have been able to keep slavery itself in place.
Bottom line is that we want to maximize freedom for individuals and small group decision making wherever possible. Larger entities should only step in when individuals or small groups can't work it out for themselves.