The other side of that coin (which is far more prevalent historically) is that speech restrictions are used to stifle dissent and repress minority groups. I.e. illegalizing anti-war or minority rights speech.

You make the mistake of assuming that those in power will only be preventing speech you agree with rather than muzzling you.