We joke about this now but all it would take is a developer or LLM operator with a dark sense of humor to trigger violent or self harming reactions in people that are already unstable.
By the time the UK was voting for the Investigatory Powers Act*, I realised that laser microphones could be mass-produced so cheaply that London's Metropolitan Police could put every window in London under surveillance for less than that police force's normal annual operational budget.
And thanks to what Snowden showed, I am distrustful of modern software. And not just closed source stuff, I expect agencies to hire people to attempt to sneak backdoors into OSS, and some of them to even succeed.
* https://en.wikipedia.org/wiki/Investigatory_Powers_Act_2016