Yes because everyone has the same morals and follows them exactly. You're basically arguing that we don't need laws if everyone just recognized wrong from right.
There are universals, for the times when there is not, you MUST recognize that by forcing them you ONLY cause harm. There's no arguing that.