>All speech besides threats of violence needs to be free, or we can't progress as a society.

The problem with that simple rule is that there is no simple way to distinguish threats. Take a statement like "I don't think black people deserve to live", technically not worded as a threat, but the sentiment is no different. Similar examples varying in strength, obfuscation and contextual meaning can be produced in unlimited quantities. No hard rule can deal with all this, so have to debate what should and should not be allowed.