What this exploit describes is not unreliable output, it's the LLM making web requests exfiltrating the user's data. The user doesn't have to do anything with the LLM's output in order for this to occur, the LLM does this on its own.
The LLM is a completely stateless machine that is only driven by input the user fully controls. It doesn't do anything on its own.
It's like the user running a random .exe from the Internet. Wow much exploit.
I'm not sure if you're talking about the initial attack vector that plants the attack in the LLM's persistent memory, or if you're talking about subsequent interactions with the LLM.
The initial attack vector may be a web request the LLM does as a result of the user's prompt, but it does not necessarily have to be. It could also be the user asking the LLM to summarize last week's email, for example.
Subsequent interactions with the LLM will then make the request regardless of what the user actually requests the LLM to do.
"The LLM is a completely stateless machine"
In this case, the problem is that the LLM is not stateless. It has a persistent memory.
If you have decided to give a 3rd party control over your LLM context, that's on you. Of course the 3rd party has as much control over the LLM as you do.
It's literally the same thing as running a random .exe from the internet. Of course this can be useful, the .exe could provide a useful function, alternatively it could also steal your data. But you chose to run the .exe. Similarly automating your LLM context generation can be useful, but with exactly the same caveats, whoever influences your LLM context controls the LLM. If you enable persistent memory you give them this control.
The LLM we are discussing here does have persistent memory, because OpenAI gave it persistent memory.
"It's literally the same thing as running a random .exe from the internet"
I'm not sure what the point is you're making with that, since downloading a random .exe from the Internet is clearly a security issue. By your own analogy, this is also a security issue. The difference is that OpenAI is doing it for you, you're just using OpenAI's program in the way it was intended to be used.
* most people will find it surprising that showing a photo from the internet to ChatGPT is as unsafe as opening a random, untrusted exe.
* many people don't even understand that it's unsafe to open random, untrusted exes.
Are you seriously suggesting that we should leave all these people to the wolves, because they're less knowledgeable about security vulnerabilities than you?
Which users do incessantly, necessitating an entire security infrastructure to combat it.