Also, again, what about speech that only indirectly affects you?
What if my speech is "I do not think gay people deserve the same human rights as straight people, and I want the government to deny them those rights"? I'm not saying it to you, I'm saying it to my elected representative(s) in the government. Should we tolerate that speech?
Additionally, I believe that history supports my position. Generally, increased tolerance seems to correlate with when society stops accepting and starts shaming people with 'bad opinions' (racism, sexism, etc).