Except you've chosen to violate the owners right to happiness by attempting to place limitations on the rights that were so called inalienable. What you think the limit should be and what the slave owner thinks the limit should be differ.
Same right, but in this case brought into conflict by disagreement of interpretations.
Second, what happens when the other side refuses to stop because he believes that your interpretation is wrong? What do you do then?