As an American, I have the right to bear arms (ok the constitution actually says not to infringe on "a well-regulated militia" but bear with me), but not the right to spray automatic gunfire wherever I please.
I disagree with your assertion that "our society doesn't view free speech as a consistent principle." Of course we do. Free expression is a staple of Western civilization, and shows us how far we have come from mandated religion and the sanctity of the nobility.
However, free expression can be damaging in certain circumstances. It is a testament to the strength of our civilization and our values that we find ways to mitigate the harm from certain kinds of expression, rather than scrapping the 1st Amendment altogether. It lasted over 200 years, why not 200 more?