Yet you and the others presenting this argument conveniently forget that not all white men were free and privileged - and that these progressive thinkers of the day, naturally strived to free and uplift those that were closest to them and mostly visible to them.
Also you forget that not all people of colour were slaves and that not all slaves were of colour. You also ignore the fact that most slaves were bought and purchased by non whites.
Applying modern zeitgeist and morals to different historic periods is wrong at best.