But the thing is, defamation would be legal under absolute free speech. Nobody could sue for it.
"Go lynch this man" is speech, but is not protected under the First Amendment. And the First Amendment gives much broader protection than most countries' legal systems do.