Isn't there a quote somewhere that says "Free speech is meant to protect unpopular speech. Popular speech, by definition, needs no protection"? "I disapprove of what you say, but I will defend to the death your right to say it"?
If you are not for the free speech of people you disagree with, then you are simply just not for free speech.