Because physicality is what defines if people are free or not, and freedom is the only moral principle that a reasonable, moral society can be based upon. If you forsake freedom, you might as well forget about morality altogether.

Regulating emotions is arbitrary. Maybe it makes me sad that you post comments on HN. Should your comments be regulated?.

Physical harm is where you draw the line, because it's the only line that makes sense.