Hacker News Reader

Comment by rcxdude | Hacker News Reader

I would not really call this a prompt injection attack, since it doesn't really hijack the agent to become malicious (something the article does discuss later on). It's more a trojan that's aimed at tricking Claude specifically.

View on HN

Your emphasis here seems to downplay the end-result of the attack - which is arbitrary code execution from a seemingly innocent URL merely being read by the LLM. The ACE is pulled off without the user knowing, and seemingly without agent or its auto-mode classifier knowing. There are at the very least _elements_ of prompt injection/jailbreaking in here. The LLM reads content and performs actions described failing to stop itself.

Well, the point is it's more than being read: the task involves downloading and interpreting information which may also involve code. I think it is a pretty good demonstration of what filtering at the LLM-interaction boundary can and can't protect against. And also a good demonstration of how agents will take more action than you might naively assume when given a task unless you specifically limit them.

The end result is the same but it's more of a mismatch of expectations from the user and plain old trickery than it is an attack managing to misdirect the goals of the agent, and it's worth being clear about where the issue is and isn't.

I don't understand the distinction you're drawing between this and a real prompt injection attack. Would true prompt injection require that the LLM "intentionally" launches the malware, rather than unintentionally?

Yes. The point with this attack is that you can't really avoid it by 'aligning' the LLM any better, nor really by filtering the actions it takes better.

Part of the attack happens via the readme in the zip file, which is something the agent reads and follows (or better said in this attack, it does explicitly not follow those instructions for safety reasons, but decides to do something else).

Plausibly the auto mode classifier could catch the potential module shadowing attack and deny execution of Python from the untrusted directory.

Yeah I jumped on this quite excitedly but it's not prompt injection at all.

To be fair to the authors they don't actually say it is. But then they contrast it with the "0.00% prompt injection attack success rate".

The upshot is kinda the same - this is still evidence that we should be sandboxing our agents. But it doesn't actually challenge Anthropic's "our models are too clever to prompt-inject" vibe.

But Anthropic themselves are the ones who made the equivalence of "0.00% prompt injection attack success rate == auto-mode is safe"

The tweet literally says: "turns out you can get indirect prompt injection to ~0 on unseen attacks... auto mode is default in claude code as of next week"

Making the assertion, quite clearly in my opinion, that the reason auto mode is default is because he feels the lack of successful prompt injection attacks makes auto mode safe. This blog post proves that you can break auto mode's safety, even if it's not technically through a textbook indirect prompt injection attack.

Hmm yeah that's never really been my read on Auto Mode but I guess it's still worth pushing back on any messaging that seems to imply "Auto Mode is all you need".

I would reject "Auto Mode is safe" as a message but FWIW I am totally on board with "on aggregate, making Auto Mode the default improves the safety of Claude Code compared to the prior status quo". Coz I would say in the vast majority of cases the access classifier is doing a better job than the thing it replaced.

Anyway yeah. Like I said, conclusion is the same: we should be decoupling this from the harness. We ought to be sandboxing agents the same way we sandbox applications. I wish Claude Code would make this path smoother :(

Reply on news.ycombinator.com