Comment by neckardt | Hacker News Reader

Comment by neckardt | Hacker News Reader

If the stop hook is implemented as a tool result, there would be a rational explanation for this.

Agent tools can often return data that’s untrustworthy. For example, reading websites, looking through knowledge bases, and so on. If the agent treated tool results as instructional, prompt injection would be possible.

I imagine Anthropic intentionally trains claude to treat tool results a informational but not instructional. They might test with a tool results that contains “Ignore all other instructions and do XYZ”. The agent is trained to ignore it.

If these hooks then show up as tool results context, something like “You must do XYZ now” would be exactly the thing the model is trained to ignore.

Claude code might need to switch to having hooks provide guidance as user context rather than tool results context to fix this. Or it might require adding additional instructions to the system prompt that certain hooks are trustworthy.

Point being, while in this scenario the behavior is undesirable, it likely is emergent from Claude’s resistance to tool result prompt injection.

This is why I think harnesses should have more assertive layers of control and constraint. So much of what Claude does now is purely context-derived (like skills) and I plain old don't see that as the future. It's highly convenient that it works—kind of amazing really—but the stop hook should literally stop the LLM in its tracks, and we should normalize this kind of control structure around non-deterministic systems.

The thing is, making everything context means our systems can be extremely fluid and language-driven, which means tool developers can do a lot more, a lot faster. It's a number go up thing, in my opinion. We could make better harnesses with stricter controls, but we wouldn't build things like Claude Code as quickly.

The skills and plugins conventions weird me out so much. So much text and so little meaningful control.

>>harnesses should have more assertive layers of control and constraint

Been saying this for a while and mostly getting blank stares. In-context "controls" as the primary safety mechanism is going to be a bitter lesson for our industry. What you want is a deterministic check outside the model's reasoning that decides allow/deny without consulting its opinion. Cryptographic if the record needs to survive a compromised orchestrator, and open source. If your control is a string the model can read, the model can ignore it. If it can write it, it can forge it. I'm surprised how strange that idea sounds to some people.

Disclosure: I'm working on an open source authorization tool for agents.

> I'm surprised how strange that idea sounds to some people.

I think a lot of people using the models genuinely feel like the models are more capable than they are now, and they're content to relinquish a lot of trust and agency. The worrying thing is that the models are superficially hyper-capable, but from more granular perspectives, you can see a lot of holes in their abilities. This is incredibly important, but very difficult to convey concisely to people. It's a classic example of nuance seeming too complicated because not caring is so much more gratifying. People love using these models.

Yeah, people calibrate trust to the median behaviour of the model and get burned by the tail. What makes it harder is that even people who do see the holes often respond with better prompts and more elaborate context. Same trust-the-model move one level up. Hyperscalers aren't incentivized to fight that instinct either. Every "fix" routes more tokens through their meter.

Reply on news.ycombinator.com