yeah, and the bigger problem (and why regulation is necessary for other dangerous fads) is that PEOPLE ARE STUPID and BELIEVE EVERYTHING THEY ARE TOLD. So AI grifter techbros come along and tell people about how "human" these things are, how "accurate" they are, how much they're supposed to be able to do, and we have morons like the google chap who quit because he thought an LLM was sentient (how tf do you get a job at google and still be that thick?). So the "average joe" is fire-hosed with messaging that is at the very least, inaccurate, and, if we're being hones - a LIE.
It's your "average joe" users who are turning to "ai companions" when they feel depressed and want to kill themselves, because they are under the impression that it's an understanding, sentient entity. If people all thought of LLMs as "bags of words" (ie, understand that they're just token predictors with ZERO understanding of anything), perhaps that would change.
But we can't rely on the grifters to be honest - they'd lose all that yummy yummy cash! - so I guess we need regulation, just like we do for driving cars or taking medications, or any other item from a myriad I could choose from.