Thank you for verbalising the issue, overconfidence was the risk I was seeing all along. Widely accepted, shamelessly overconfident, bus still potentially gibberish generator.
I believe such a generator can be very harmful in our society that has not yet figured a way to scalably teach critical thinking.
Harmful at societal level by flooding the world with generated content therefore drowning actual information and by creating a lot of overhead.
Harmful at the personal level by increasing the difficulty of sorting information and by creating toxic dependencies (see the Replika scandal).
Harmful at the class level by further concentrating power towards plutocracy.
Harmful at the professional level by eliminating the lower levels of the professional ladder and therefore destroying the pipeline of skill.
And harmful at the interpersonal level by casting doubt on the authenticity of relationships.
Unfortunately I don't really know of resources to point you to. These are mostly my thoughts. Most probably though influenced by a diet of sci-fi. I can however expand on them.
For somewhat recent and relevant content check The Orville S03E04. At the time it was just a critique of fake news but I think it is correct about what can happen if AI makes fake news easy to scale infinitely and with very low barriers to entry.
I think presently worrying about Sentient AI is misguided because Generative AI is equally serious and much more urgent. There is no need of further advancement. What I said is possible with current AIs already.