Almost all situations of real interest and disagreement will be grey-area and call for a pragmatic vs. a dogmatic approach.
> First of all, do not put equality sign between physical and being verbally abusive.
I am not, in general -- let's try to confine ourselves to my specific example.
> The first one is something external that you have no control over, no matter how much you fight, the second you can just shrug off and continue with your life.
This is a dogmatic assertion that is not supported by the evidence. If I were feeling less charitable, I could counter that of course you could do something about physical violence -- you just had to train harder, have better weapons, or build a coalition. People are emotionally abused. Stating that in some cases, some people can transcend that abuse is ... not really relevant. It's interesting, and should be studied so that perhaps we can confer immunity to more people.
> Problem with your example lies in the lack of context. On the more humorous side - maybe your wife is a cunt and your children are little bastards.
As soon as you admit context you admit that abusive speech could be an act of violence, as well as true speech. As you say, yourself, context matters. This is why dogmatic approaches to this issue are doomed to fail.
> If you percieve any kind of abusive expression, no matter of the context, as violence, ...
Let's just stick to my example. To disprove "speech cannot be violence", I only have to find one example of speech that is violence. I don't have believe or prove that all (abusive) speech is violence.
> then you're just done. There is no useful way that you can contribute to culture and society, because you cannot participate in discourse anymore.
This is nonsense. It's smacks of litany rather than logic. A person can be a productive member of society in many ways, yet hold beliefs that disturb you.
> Going further - your comment disgusted me and I've felt that it violates some of the most important values I have in my life.
I can't know, but your words suggest that you are looking to be disgusted. We all like to be self-righteous at times, but I doubt we are as far apart as you believe.
I have deep knowledge of computer science, artificial intelligence, biology, and emotional abuse. All of these areas teach that nature has no respect for our ontologies.
We may try to draw a hard line between violence and speech, but we mislead ourselves. Almost all of the cases worth debating occur at the fuzzy edges of these concepts.
Failure to realize this may result in the sense of having a crystalline framework, but ultimately it misleads.