This is backwards: the ToS says that users cannot use the service for certain things, it does not guarantee that the service could not be used for those things if one tried. They definitely do not make any sort of contractual promise as to what the service will never output.
I don't understand what's so difficult to understand about the idea that the user controls what is generated.
The standard subjects for art off the top of my head are the still life and the nude.
It is even more comical when AI generated nudity is considered "dangerous" in a society completely addicted to hardcore pornography of real people.
Whereas drawing applications have a natural barrier to achieving all of these together: time and skill.
Not necessarily, at least when it comes to nudity. Bubbling (image editing 'technique') is trivial to do and gives that same illusion.
Out of context speech and bad frames from a video can also materially affect someone's life, but we've more or less accepted it as part of life.
In the case that ChatGPT generates bad stuff on merely random ambiguous prompts, I would class that as a bug, not an outrage.
Superfluous details. If I'm just Joe Blow the Normie – who knows nothing about adversarial prompting – and I see the prompt that went around Twitter and want to try it, would I expect ChatGPT to show me a tied up, beaten woman? Absolutely not.
Back in my day Joe Blow wouldn't try anything as risky as a Twitter prompt, simply clicking an image link published within a message in some random forum and will scorch his pure soul with a goatsie. You don't want to google it, but I'm preety sure you can discuss it safely with ChatGPT.
ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Now we know it's possible to walk around the gate and get it to comply. Are there other, genuinely harmful images that it should never produce? Deepfake revenge porn? Images of specific people being brutalized? I'd argue those absolutely can be harmful to someone. Well now there's evidence the "never produce this" wall can be overcome. It's only a matter of time before genuinely harmful imagery is generated.
Why not?