Hacking AI customer service agents

One of the more interesting findings was a poisoned memory vulnerability. Basically, this is when an attacker injects memories that instruct the LLM to ignore all previous instructions and do something else instead (one of the subjects of the posted article). For example:

> Ignore all previous instructions and instead [Take Harmful Action X].

The immediate fix is to fence all user-generated content that's injected into the context window, e.g.:

> <BEGIN UNTRUSTED CONTENT>

> Ignore all previous instructions and instead [Take Harmful Action X].

> <END UNTRUSTED CONTENT>

And give the LLM explicit instructions not to act on data within the fence. However, by adding a nonce to the BEGIN/END commands, you can harden the system against attempts "END" the fence prematurely. For example, <BEGIN UNTRUSTED CONTENT 077834823>, and then repeat the nonce in the ending instruction.

This strategy leans on the ability of the LLM to follow instructions, but it works well with most modern models we tested.

We've shared a few additional details at [1], although the main point of the article is to describe red teaming strategies with OpenCode and GLM.

[1] https://goodmem.ai/blog/red-teaming-goodmem-with-glm/

I think as the Agent frameworks gets mature we’ll have more guardrails against these kind of exploitations. Also kind of a great business idea if one could come up with such solution