It does cut both ways, but that doesn't justify your "let people say what they want no matter how awful it is" conclusion.

At the margins it's certainly hard to tell sort-of-bad content from not-so-bad content. But that doesn't mean we can't find pretty clear examples. Banning child pornography, for example, doesn't seem to have harmed freedom of speech much.

You also ignore that what speech we allow influences what people will say at all. For example, if we allow violent threats and harassment (which are both definitely speech), this will be used to silence voices. And in the American context, it's pretty clear which voices get silenced. Which voices have been silenced for decades and decades.

So in practice, somebody will always be silenced on some topics. The question is more who gets silenced, and what kind of environment for discussion we want to create.