Height (and the other attributes that go with it) generally confer an advantage in a physical fight.
Or it is simply that you can physically intimidate people to get an upper hand in situations where such an upper-hand shouldn't be given... just because. How noble. What a useful and admirable trait to encourage.
There is that deterrent in place, that social constract. Sure it can be violated, but your argument is along a similar line of folks who advocate for open carry.
And if you are killed? There is a strong appeal to reducing the likelihood you will die.