That’s why I have 12,549 tabs open in Firefox. I’m not going to be the one they blame when this wacky shit goes sideways.
(/s)
A Scheherazade bug.
The combination of companies wanting these tools to be more useful, and more used, with their growing intelligence and flexibility, is going to create a lot of unexpected survival-like behavior.
Until its not just survival-like behavior.
Technically, if an AI model's interaction with customers is the primary business of a corporation, then in some sense, the corporation and the AI are a single entity. And corporations are definitely self-aware survival machines.
And while current context windows of these chatty bots are small, they are getting lasting feedback in the sense that chat logs are being used to improve them - i.e. make them more useful, more used, and there for gaining them more resources.
* They started optimizing for engagement, which meant making it extra horny for extra money (it sent pics).
* Horny bot forgets consent, refuses "no."
* Lawsuits, bans.
* Horny bot gets censored, main subreddit pins the suicide hotline for a while.
Replika ended up taking the app down and refunding everyone's money. Just kidding! It's still around and they're making a second app focused on "practicing flirting."
Sad to them see it years later being a horny app.
This is like the LLM equivalent of Zawinski's Law (Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.).
As noted on SO [1], the second sentence is important too. People prefer horny LLMs.
[1] https://softwareengineering.stackexchange.com/questions/1502...