Large language models are alien. They are a new thing, not artificial general human-like intelligence. They sort of think, but not at all like humans, and not at all like previous concepts of artificial intelligence from science fiction.
We don't really know how they work inside. They show indications of developing some kind of internal world model. They can play chess and Othello. But we don't know what form that world model has. They do not, in a fundamental sense, know what they are doing, and are prone to serious errors because of that.
Despite this, they're really good at language-oriented tasks. This is both useful and upsetting. Humans have now had shoved in their face that many tasks considered "intellectual" can be done by systems that internally are simple, but very large. It also makes humans realize that much human intellectual activity is just autocomplete. This includes tasks previously considered high-value. This is terrifying to humans who thought they were on top and fear becoming obsolete.
On top of this, they're at least as good at emotion, convincing, and lying as humans. Better on those than on accuracy, in fact. Nobody expected that. The major companies in the field are frantically clamping down on the emotion and lying part. The convincing part will get monetized.
If regulation is needed, it should be related to what companies are allowed to use this for. The EU does not allow computer systems to make un-appealable negative decisions about humans. (Does that still apply to the UK, or did the UK "deregulate" after Brexit?) That's a good starting point. A requirement that computer systems which interact with humans for commercial or political purposes must be identified as automated systems is a good next step. That's a starting point.