I imagine that these apps are a valuable target for scammers to hack. People who are open to manipulation/affirmations from an AI are just a step away from victims of romance scams.
“Drink Mountain Dew, my love, you’ll feel better.” - is that the end goal?
https://apnews.com/article/chatbot-ai-lawsuit-suicide-teen-a...