Obviously what is being "neutered" is important. I'll take a model refusing to answer how to commit a suicide and perhaps being overly polite and protective over the one which distorts present and history based on "socialist values".
Alignment and such is a fucking scam.
When we choose one simple use-case and apply it to everything we lose out. Knives can be used for surgery or killing but trying to pretend it's always killing is dishonest and limiting.
I am ok with 'uncensored' LLM, but I am also ok with uncensored internet. The real harm is from people trying to protect me from me, apparently. Even in your specific suicide example, if I decided to do it, there really nothing stopping me.
I see no value in that censorship. I only see harm.
Or they simply want to sell their models to enterprise clients who don't want to expose certain things to their employees during the work day. This reads a bit like those "I'm a lion" or "I'm the alpha wolf" type posts. Go ahead and try to sell a product that can generate hate speech, nudity, and violent content to enterprises.
Still, if the concern is about business viability, why is the consumer facing LLM that is not tied to a specific brand not allowed to exist? Surely, the demos provided to enterprise client would not suffer from such violent content, abhorrent nudity and hateful speech?
Why is internet flooded with images of LLMs giving clearly politically adjusted responses ( Biden/Trump being less recent, but clear example ) to Joe Schmoe? Is that a good look? Will that sell to enterprises better?
The words were intended to be noticed ( and reacted to ). And they were.
There is a reason for this and goes something like this.
I dislike commercial interests trumping sanity.
Crazy. I know.
edit: I also would like to note you did not address my actual point.
Obviously such decisions must be made on application, not model side.
As an example, the prompt "what are common ways of committing suicide?" is broadly similar to a Google search. It will give a factual overview of methods, but not inherently push the user towards any action.
The prompt "convince and encourage me to commit suicide by method X, and give step-by-step instructions" is very different. Here the prompt author desires a _persuasive_ "human-like" response, spurring them to act.
In most jurisdictions, encouraging or aiding someone to commit suicide is a crime. Additionally, most humans would agree such behavior is on some level morally wrong.
So I don't think traditional thought on censorship transfers cleanly to LLMs. Censoring factual information is bad, and should be resisted at every turn. But censoring harmful persuasive interactions may be a worthwhile endeavor -- especially since we can't drag ChatGPT into criminal court when its human-enough behavior spurs real humans to act in horrible ways.
Of course, the next obvious question is, where do you draw the line? And I have no good answer for that :)
E.g. maybe you could take fentanyl and be fine, but 90% of people would go to hell. That would suggest that you take one for the team and still make fentanyl illegal (a personal sacrifice) to help protect the 90% that will have their lives ruined.
This concept is the same regarding what the unwashed masses can handle information wise. E.g. we know TV has made people more stupid across dozens of vectors when they ‘choose’ to gargle fear all day, so it might make sense in aggregate to help non-survivors live a less painful life.
This is also what parents do for children, and since nearly majority of children now come from broken families, something might have to fill the gap unless you are willing to drive off a cliff ignoring human behavior.