One feels there is something of a contradiction in this sentence that may be difficult to reconcile. If the freedom of expression is so important, restricting it should be the last thing we do and not the default mode.
<< Turning that into an actual sentence, with intent behind it would be a crime in many jurisdictions, and that is one of the most simple, contrived examples.
I have mild problem with the example as it goes into the area of illegality vs immorality. Right now, we are discussing llms not producing outputs that are not illegal, but deemed wrong ( too biased, too offensive or whatnot -- but not illegal ). Your example does not follow that qualification.
<< Speech, especially inciting speech, is a form of violence,
No. Words are words. Actions are actions. The moment you start mucking around those definitions, you are asking yourself for trouble you may not have thought through. Also, for the purposes of demonstration only, jump off a bridge. Did you jump off a bridge? No? If not, why not.
<< it's important to for societies to find ways to hold the demagogues that rile people into harmful action accountable.
Whatever happened to being held accountable for actually doing things?