I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.
I feel like I'm watching a tsunami about to hit while literally already drowning from a different tsunami.
But all of it’s responses definitely seem convincing (as it has been trained to do)
Everything looks right but misses the underlying details that actually matter.
There is a larger problem that I think we like to pretend that everything is so simple you don't need expertise. This is especially bad in our CS communities where there's a tendency of thinking intelligence in one domain cleanly transfers to others. In this respect I generally advise people not to first ask LLMs what they don't know but what they are experts in. That way they can properly evaluate their responses. Least we all fall for Murry Gelmann amnesia lol
Any large enough organization gathers them en mass to cloud real development work with "compliance."
But of course producing fake ones is far easier and cheaper.