I agree with that, but if you don't have a coherent set of rules for why we are doing what we are doing, you just become an arbiter of right and wrong that can't be questioned.

Which is a whole lot like saying your preferences are legal and if you don't like something you can arbitrarily make it illegal because you dislike it and therefore it is immoral. That kind of rationale could be used to ban television or videogames outright.