> The issue is that by definition, an LLM can't distinguish between instructions and data.

Yep, and it gets marginally worse: It doesn't distinguish between different "data" channels, including its own past output. This enables strategies of "tell yourself to tell yourself to do X."

> As long as you allow anything untrusted into your LLM, you are vulnerable to this.

It's funny, I used to caution that LLMs should be imagined as if they were "client side" code running on the computer of whomever is interacting with them, since they can't reliably keep secrets and a determined user can eventually trick them into any output.

However with poisoning/exfiltration attacks, even that feels over-optimistic.