Take that phrase to its logical extent. Should certain speech mean imprisonment? Should it mean execution by the government? Alienation and banishment from society? Attacked and lynched by a mob? What about political affiliation? Philosphical or religious beliefs? Those are all consequences that have, and are used against people who exercise their natural rights. Those are the natural outcomes of limiting free speech. Limiting free speech is the end of a free, liberal society.