Well, they might still be people in a position to significantly affect the quality of the slave' lives. Which should not be conflated with "slave owner".