It is very easy to use it to harm OpenAI, and that is enough for OpenAI to wish to constrain it. You cause it to write something objectionable and then publish it to reap all the attention and ad-revenue which goes with people being outraged. Best if you manage to provoke into outrage inducing output without your prompt itself being outrageous, but not strictly necessary. During this OpenAI’s reputation would be harmed.
This is not even hypothetical. There was that microsoft chatbot which “turned nazi”.[1] They then had to turn it off.
Did it do many other interesting things? Probably. Yet the few problematic outputs is what we remember it by.
1: https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...
At the time, I thought it was incredible that Microsoft was held responsible for this! so much so that they had to turn it off because of all the blame. It was really weird and it made no sense. Here we are again.
I hope ChatGPT is put to better use.