1) Certain speech such as "shouting fire in a crowded theatre" has long been considered illegal.
2) The US has much more expansive laws about "conspiracy to..." which make it illegal to have certain conversations.
Oh, those don't count? Not real speech. Sure, why not.
3) Most people will experience very real economic consequences for a very wide range of speech. We're not all yeoman farmers entirely independent of paid employment.
That presumably doesn't count because only governments can restrict liberties? Well, have it your way but I would regard losing my job due to speech as a pretty severe restriction on my liberty.