Harmful? Meh. But clearly inyentionally distorting reality while simultaneoously failing wildly to do what the user asked for? Yes. And that's a bad precedent for a tool we hope will help us be smarter.
Distortion and confabulation must be expected, but how can we train the public to properly interpret the responses?
if there is no truthiness to it, even in part, then what use is the tool? outside of brainstorming some creative media ideas, why would I ever use it for practical work purposes where I need a correct answer?
plus once you shatter the illusion of trust, truth, and validity, all outputs may now be suspect. what else is getting nudged? subtle biases will certainly creep in, but if blatantly obvious or true data can't be delivered, why would I trust any other output?
The idea of asking for an image a Confederate battalion and getting a Benetton ad is a joke to most people but there's some minority who won't get what's funny about it. The longer this keeps up the fewer people will get the joke.
If you ever see pictures of how people in the middle ages drew things in the classical age you can get a feel for - they just drew pictures of everyone as if they were contemporaneous.
The question in my mind is if abusing the truth like this is a greater harm then the benefit that comes from winning this political argument. It's like something out of 1984 I know but ... what if it is _really_ is a good thing?
I feel like it's something out of fiction you know? Like The Ones Who Walk Away from Omelas - you can have a perfect society that just depend about telling this one set of lies. Or ... there was some other sci-fi story where it turned out that the previous generation had committed genocide and then covered it up by a sort of consensus of silence and shame.
Realistically it's a fool's way around whatever problems we have. I can't any good comes out of telling yourself a lie. Plus it will and would never work. But still. You have to wonder...