Already there are stories of these AI companions encouraging people to kill themselves, sometimes to the point of actually doing it

https://apnews.com/article/chatbot-ai-lawsuit-suicide-teen-a...