Simple: A man's rights end where another man's rights begin.

To use your example, the slave owner's right to pursuit of happiness ends where the slaves' rights to liberty and pursuit of happiness begin. A would-be slave owner cannot and should not violate another man's (a would-be slave's) right to liberty and pursuit of happiness.

Going back to the subject of marriage, my right to liberty ends where my would-be wife's right to liberty begins and vice versa. Marriage is inevitably a compromise of both our rights to liberty. Thus, I find marriage a violation of human rights.