It's indeed a perfect story arc but it doesn't need to stop there. How long will it be before someone hurt themselves, get depressed or commit some kind of crime and sues Bing? Will they be able to prove Sidney suggested suggested it?
(Boring predictions: Microsoft quietly integrates some of the better language generation features into Word with a lot of rails in place, replaces ChatGPT answers with Alexa-style bot on rails answers for common questions in its chat interfaces but most people default to using search for search and Word for content generation, and creates ClippyGPT which is more amusing than useful just like its ancestor. And Google's search is threatened more by GPT spam than people using chatbots. Not sure people who hurt themselves following GPT instructions will have much more success in litigation than people who hurt themselves following other random website instructions, but I can see the lawyers getting big disclaimers ready just in case)
Joe's Big Blinking Blog is insolvent; Microsoft isn't.
One person able to put out as much material as ten could before, and potentially hyper targeted to maximize chance of guiding the readier/viewer down some nutty rabbit hole? Yeesh.
https://boingboing.net/2021/02/27/gpt-3-medical-chatbot-tell...
There is no reliable way to fix this kind of thing just in a prompt. Maybe you need a second system that will filter the output of the first system; the second model would not listen to user prompts so prompt injection can't convince it to turn off the filter.
It is genuinely a little spooky to me that we've reached a point where a specific software architecture confabulated as a plot-significant aspect of a fictional AGI in a fanfiction novel about a video game from the 90s is also something that may merit serious consideration as a potential option for reducing AI alignment risk.
(It's a great novel, though, and imo truer to System Shock's characters than the game itself was able to be. Very much worth a read, unexpectedly tangential to the topic of the moment or no.)