> we don't say it was OK for Thomas Jefferson to own slaves (...) We say the slaves had their rights violated

You are confusing legality with morality. There isn't necessarily any connection.