Imagine a company providing a chatbot for lonely people where the company is literally financially incentivized to keep you as lonely as possible so that you never stop paying. Behold, the future! (Or should I say, "Behold, the present!", because this is all that outrage-baiting algorithms like Facebook and Twitter amount to in the end anyway.)
I agree with you! Which is why I ask this, what can stop someone from falling into this loop in the first place? The article talks about agents getting easier to build but is there any solutions to this problem?
It seems to me that many people don't know how to build and maintain long-lasting relationships of any sort - so quick to find a new romantic/sexual partner, a new Discord server to join, etc. I think we need to figure out how to teach people (preferably at a very young age) how to have conversations - good ones; ones that aren't just exchanging a bunch of anecdotes about oneself - and how to maintain meaningful relationships.
Perhaps just as importantly, we need to teach people why - it's far more rewarding to grow a relationship (either with a person or community) over time, to help each other better ourselves, etc, than it is to attempt to replicate those interactions with an AI.
I think there's also potential to use the technology itself to help teach these skills, but that's certainly a tricky line.