I hate generic statements like, "if you use speech to try to hurt people". They tend to be made to let you say something and get many people agreeing with you who would not otherwise if they knew what you really were after. So I would ask you to define what constitutes "hurt" and define what constitutes "people" in your sentence.
I don't think you mean all people... because in the same sentence you advocate harm (or the de facto threat thereof) to some people based on what they might say. So is this hurt feelings? Actual physical harm (or the fear thereof)?
Or are you suggesting that people should be free only to say those things within some pre-approved range of ideas of which people like-minded to yourself are prepared to tolerate?
To be specific, if you're saying that any person that would advocate physical harm, or cause a reasonable person to be fearful of physical harm, should be prevented from speaking... then I'm likely with you. If you really mean more than that, then "hell no!" back at ya.