When stated this way, at least so directly, it's not surprising to me that more people aren't right-libertarians. Of course, when you have hard and fast principles, and trade only in 'thick' concepts, then you can get exploitable concepts (as you rightly point out), but we have institutions for dealing with that - the legislature and the judiciary, and there need not be a law that applies (practically speaking) in every circumstance. In society, we don't have to always be on a slippery slope.
Here's a good paper on the vacuity of the pure harm principle, as you seem to espouse, in lawmaking: https://digital.sandiego.edu/lwps_public/art17/
>Fear, as opposed to outcomes, does not need to be rational or grounded in reality. Once you acknowledge a right to be free from fear, you are able to justify any tradeoff. I say - no!
The core question is whose reality? That of the victim or the perpetrator? However, if you'd like to talk about reality, it's not at all difficult to imagine psychological harms or even PTSD triggered by something as 'unreal' as fear, which has very real outcomes. Every industrialized country in the world has some legislation against, for example, sending threatening letters, despite the fact that their only outcome is 'fear'. This is usually known as a law against assault. The dichotomy between 'mental harms' and 'physical harms' with regard to the law (and 1A jurisprudence) is on extremely shaky metaphysical grounds[0]:
>An immunization I choose to have may (momentarily) hurt me physically, but it does not constitute a harm if I have chosen to be immunized. And defamatory statements that damage my reputation may harm me even if they cause me no physical or even psychological injury (as in the case in which I don't find out about them).
[0] http://susanbrison.com/files/B.16.-speech_harm_and_the_mindb...