Really great food for thought here, I'm going to have linger on it some more. I don't know enough about LLMs to know how to counteract hallucinations, but I wonder if you could construct an LLM against a vetted corpus (much like Anthropic does with Claude) in which it solves for the problem of generating robotic speech patterns. I do agree with you a lot on the "sound" problem because I don't think I'd advise a non-native English speaker to blindly trust ChatGPT as a learning mechanism based on my experience of using it.

So, in conclusion, I'm curious to learn whether this is a solvable problem or if LLMs are inherently not the right tool to use.