The statement that free speech is a right is, itself, an ethical statement. Claiming that ethics is arbitrary is basically saying that free speech is inviolable is an arbitrary claim that holds no weight. I'm constantly annoyed by the unfounded post-modernist claims of arbitrariness and subjectivity in ethics. The truth is that ethicality is a "Hard Problem" of philosophy (also an oft-devalued field, but one without which there would be no math or science).
I agree with both you and Voltaire, that freedom of speech is an important right that is "inviolable". That is, it posible to infringe that right, but doing so is wrong, and infringement of free expression is ethically subject to defense against the aggression represented by that infringement. In short, even if you don't know what the fuck you're talking about with your use of the word "ethics", and even if you contradict yourself by saying that free speech is a right (an ethical claim) and that ethics "are totally arbitrary", I fully agree with your basic assertion that free speech is a right, and anyone who wants to claim otherwise had better be able to provide a valid, deductive argument from apodictisms (or at least agreeable axioms) -- especially because I can provide such an argument in support of free speech -- before I'll give that claim the time of day.
Just to restate, in brief:
1. Yes, you're absolutely correct in your statement that free speech is a right, and it should be defended, and not deprecated by claims it's a "privilege". Yes, there is responsibility associated with that right, but that doesn't make the right into a "privilege".
2. Your statement that free speech is a right is an ethical claim, so maybe you should re-examine your dismissal of ethics. Thanks in advance for thinking it through.