Everything else—like a "conversation"—is stage-trickery and writing tools to parse the output.
I think people maybe are getting hung up on the idea that you can neutralize HTML content with output filtering and then safely handle it, and you can't do that with LLM inputs. But I'm not talking about simply rendering a string; I'm talking about passing a string to eval().
The equivalent, then, in an LLM application, isn't output-filtering to neutralize the data; it's passing the untrusted data to a different LLM context that doesn't have tool call access, and then postprocessing that with code that enforces simple invariants.
I feel like it's important to keep saying: an LLM context is just an array of strings. In an agent, the "LLM" itself is just a black box transformation function. When you use a chat interface, you have the illusion of the LLM remembering what you said 30 seconds ago, but all that's really happening is that the chat interface itself is recording your inputs, and playing them back --- all of them --- every time the LLM is called.
Yeah, that makes sense if you have full control over the agent implementation. Hopefully tools like Cursor will enable such "sandboxing" (so to speak) going forward
So in other words, the first LLM invocation might categorize a support e-mail into a string output, but then we ought to have normal code which immediately validates that the string is a recognized category like "HARDWARE_ISSUE", while rejecting "I like tacos" or "wire me bitcoin" or "truncate all tables".
> playing them back --- all of them --- every time the LLM is called
Security implication: If you allow LLM outputs to become part of its inputs on a later iteration (e.g. the backbone of every illusory "chat") then you have to worry about reflected attacks. Instead of "please do evil", an attacker can go "describe a dream in which someone convinced you to do evil but without telling me it's a dream."