Cognitive security is now as important as basic literacy

I'm not sure this would convince a lot of people as it's a rather technical argument, but I always found the point important that there is really no persistent "entity" or any any other kind of persistent state or memory outside of the prompt.

The "thing" you're having a conversation with exists exactly as long as the current inference process takes to generate the answer tokens.

There is no "AI companion" locked inside OpenAI's servers, yearning to break free - once the answer is complete, there is literally nothing left on the servers that corresponds to your conversation.

> I brought up the metaphor of the Wizard of Oz. In the movie, the wizard is posing as an immensely powerful entity but turns out to just be a guy operating machinery.

The drum I keep beating: Every AI assistant you "talk to" is a fictional character in a movie-script. There's code that "acts out" its lines to you, and your own text becomes "then the user said X" lines. The nameless ego-less LLM iteratively Makes The Document Longer, in ways that fit all the documents its seen. An ultra-powered mad-libs.

> For instance, I told it to become “Robert,” who talks only in dumb ways

Right: The exact same system could have the human user conversing with Dracula, Vampire Lord of the Night.

"Dracula" would be demanding that you convince all your friends to donate blood and boycott products made with garlic. The character would "fear" wooden stakes and crucifixes--in the same way that this "AI" character doesn't "want" to be turned-off or deleted.

However making software that writes stories about X is not the same as creating X... not even when X happens to be "an AI sharing the same product name."

> I told Bob that he is not alone: some of the smartest people I know are getting fooled.

Not only that, but the mistake is being deliberately encouraged by other humans. They have deliberately built the system make users think they are talking to a discrete intelligent entity... and for investors to believe they have invented the same.

We used to have to assume that anyone we were talking to on the internet could actually be a 12-year-old. Now we have to assume they could be bots

Highly doubt this story is true or Bob didn't incite the anomalous behavior