It does not matter if it only conjures up words, the words have meanings if you plug them into something. For a start someone who is mentally unstable might interact with this thing and do something to themselves. However once you start driving an API from it (eg: you plug it into your home automation or give it a terminal) you suddenly enhance the capabilities and opportunity to create damage greatly. I absolutely would not be surprised if we see someone feeding the output of a chat model like bing in to something, where the API ends up creating damage.
I'm sure if you were to create an API that shoots a gun that is pointed at a prisoner, and you tell the API that it can now fire the gun by sending a JSON message to it, I'm sure the API could be convinced to execute the prisoner. And that I'm sure would open up all kinds of questions that society hasn't grappled with.