Handbook.md shows that long policy documents do not reliably govern agents
arxiv.org
arxiv.org
Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely continue.
Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
Ah yes, magic that costs the same as a new car.
If you have a semi-recent MacBook with even 32GB, you can run 20GB models that are pretty damn smart, with room to spare for the rest of your toolchain.
If you’re reasonably connected to the code you’re writing and prompting the AI at the level of the code, not the level of the feature, you can get some fantastic results.
Sure, it’s not the completely automated dreamland that’s been sold, but it’s still a speed-up on par from going from assembler to a higher-level language, which is still immense.
And for effectively free, if you have a machine that would otherwise have been considered “developer-grade” for a lot of tasks anyway.
If you set a large context window you end up using all your memory. Prefill is slow. And the 10-20B models are somewhat poor at understanding intent and tend to think in circles, so if you set a small thinking budget to control context size the model ends up answering based on some half formed thoughts.
Or something like batch document summarization or image classification or whatever yeah it's great. But for "agentic" and coding workflows I think you would need a completely different harness setup and prompting style compared to what we are used to from the big proprietary models. I certainly have not yet figured out what that is.
The recall text was a simple hash generated, filler text from a dictionary file and a request at the end to return only the hash from the beginning of the prompt. Past 32k characters the response contained hallucinations of characters or full hashes.
Just having large context size doesn't paint a full picture of capabilities, prompt adherence and other quality metrics.
I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things to lose focus to (or, mismanage the limited attention heads resources - per token)
After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ...
... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then any tokens in context it can associate with Paris will get a greater share. If the word Paris is in your context, it might get 0.3 or 0.4 weight, and close by tokens might get 0.2, while other (unrelated) tokens get 0.03 or something.
Now, add lots of context, and you start to see the problem: more context = greater distribution of the attention budget. Even if the prompt is about Paris, and the Paris tokens get higher weights, they are only getting (say) 0.002 ... while unrelated tokens are getting 0.001.
All LLM "answers" are just math, computation, based on the context and those weights. If it can't "focus it's attention" because it's distributed among too much context, it's far more likely to miss the relevant tokens (eg. the Paris ones) and give you an answer that ignores key parts of context.
But again, there's no fixed number of things it can pay attention to: it's a gradual degradation of the chance of it seeing key info it should, based on the amount of context.
EDIT: Ok, I guess there are an internal thing called "attention heads"; TIL. They work exactly as I described (essentially I described a single "head").
Still, what I just wrote remains true: whether you have 1 head or 96 or ... however many your particular model has ... the fundamental issue isn't the number of heads. It's not like the LLM can pay attention to exactly (say) 96 things at once: each one of those heads has all of the available context "competing" for attention as described, and then the heads themselves "compete" to produce the answer (although not through the same mechanism; it's more like the weighting between heads is baked-into the LLM).
At the end of the day, the problem isn't "there aren't enough heads to pay attention" and the LLM hits a fixed limit. The problem is each head is trying to weight the relevant stuff, and the more stuff there is to distribute weights through, the less likely it will get it right (and then when all the heads are combined to produce the final answer, it misses key context).
Yes there quite literally is internally in an LLM.
Just because something is called an "attention head" doesn't mean the terminology makes sense.
If I'm reading the article right, hungryhobbit is accurately describing a single attention head.
And... what having multiple attention heads means is that you do the "single attention head" thing several times, and average the results. There is no part of hungryhobbit's comment that changes if you make this adjustment.
In particular, he is clearly responding to this clause in the comment above:
>> the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time
Those are the "attention heads" that hungryhobbit is saying don't exist. You'll notice that this definition of "attention head" does not correspond to any part of an LLM; I fail to see the problem with saying that "attention heads", as defined by this clause, don't exist.
How was your trip to Los Angeles?
There were no movie stars!
Is the second person's statement "objectively false"?
Or is it more likely that you, Catloafdev, enjoy intentionally misinterpreting simple statements?
You might notice that in the remainder of the sentence you quoted, hungryhobbit takes the trouble to define the attention heads that don't exist.
Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"
The first 20% of context is usually very solid. Agent stays on task, is responsive and focused on the original goal.
The last 20% of context is usually pretty bad. Agent tends to loop, repeat, drift away into sub-goals (or non-goals).
The middle 60% is a bit of a grab-bag. Sometimes it's fine, sometimes it's not. It varies by task, prompt, and luck.
The above 25% and 50% limits seem like reasonable and simple rules to keep agents in the productive zones of usage.
It's generally not worth doing extensive testing to try to find your exact threshold unless you're running exactly the same prompt at scale, on similar data.
If you are in that spot, I won't actually recommend langchain tooling (I've been somewhat frustrated with it) but I think the concepts are worth understanding.
You can do something like https://docs.langchain.com/oss/python/langchain/test/evals. Trajectory evals in particular seem to suffer as context length grows (https://docs.langchain.com/oss/python/langchain/test/evals#t...).
I've used this for testing an agent that handles provisioning and permissions for developers with SaaS tooling (ex - if a developer asks, go provision an account [saml/oauth/etc] with the 3rd party, configure access, and let them know how to use it).
I'm about 6 months out of date at this point, so I won't speak for the latest models, but what was available early this year absolutely suffers from extended context lengths.
there have been some papers suggesting that the useful context is even smaller, and stays fixed as you change the context window size.
as a more general case though, i think the possibilities for what youd need to include in training to have many different paths of text be well represented enough over the long window means the later tokens will almost always be a lot more random than early ones?
there's a sheer amount of bits problem.
What matters is the relative distance. Using the latter 250k should be equivalent to using the first 250k.
It's my understanding that, for all current models having long-context capability, the "parts" of the model that allow them to do long-context processing, are mostly-untouched descendants of academic model architectures; where these academic models were designed around a very constrained use-case assumption: that anyone using more than a "normal" amount of context would specifically be trying to apply a reasonably-sized prompt to an unreasonably-sized block of plain-old embedded data, as a single-shot conversation. (Consider a task like e.g. "summarize this entire book." Context = [prompt] + [text of book].) This is the use-case all these architectures were refined and benchmarked around. And even today, this is still basically the only way these models' long-context processing capabilities work.
---
But I think it goes deeper than that. It's not just about how we've made long-context work so far. It's about what "attending to context" really means, and what that implies about what we'll ever expect a long-context model to be able to do.
I would describe long-context models as seeming to have two distinct types of "attention state" used when attending to the layer-1 input-vector / context window:
- They have some small amount (e.g. 8-32k) of regular "high-quality" attention state. This is likely inherited from the base model they are trained on top of.
- They also have a large amount of "low-quality" attention state. This is what the long-context training process has created.
The "high-quality attention state" isn't limited to referencing the first 8-32k of the context window, to be clear. The high-quality attention mechanism is just limited to "choosing" 8-32k of arbitrary data samples, each from its own arbitrary position in the context window, to form a bounded-sized vector upon which high-quality attention is then computed.
(You could think of this as the high-quality attention mechanism computing attention over the entire context, and then doing top-K sampling, i.e. zeroing out all but the K largest-valued elements of the resulting latent vector, where K=8-32k. That's not what it's actually doing—that'd require paying the very quadratic scaling cost that prevents us from just increasing the short-context window size—but it's a useful naive model for what it's doing.)
---
It's harder to directly explain what the low-quality attention mechanism does, so let me try an analogy.
(While no one can yet give a definitive accounting of the abstract computational processes the matrix-multiply-ops against a given LLM's weights encode, we can still speak of possibility-spaces, or analogies via information/computability theory on what sorts of abstract operations the LLM "must" be performing to achieve the results it achieves. This is that kind of thing.)
Think of an LLM's evolved context-window attendance mechanisms as being somewhat akin to the LLM being a regular CPU program, that has access to a 1. block of raw addressable memory (i.e. the context itself), and 2. a block of pointers into that memory (i.e. the learned first-layer attention over the context.)
Our LLM program's logic, through mutations during training, gradually evolves into a form where it could be seen as implicitly declaring and using a growing set of variables: some bound to static addresses in the block of raw memory (i.e. static context-window positions), and others bound to static addresses in the block of pointers, where the raw-memory-address-value of the pointer (and thus what data from the context-window it will read as when "dereferenced") is the result of a dynamic calculation over one or more other variables (i.e. an arbitrary weighting of the layer-1 input vector, where the resulting vector element is used as a selector, in layer 2, among other data that got sampled into "passthrough" vector-elements back in layer 1.)
Under this analogy, a model's "high-quality attention state" is the set of "named" pointer variables that it directly references in its logic.
As should be obvious, there can only be so many of these; you really fundamentally can't scale them up, without literally having "more logic" that defines and makes use of them (i.e. scaling up the size-in-weights of the model's layers.)
A model's "low-quality attention state", on the other hand, is the block of pointer memory itself, which can be as large as you like. Presuming the computational substrate permits the logic to compute on the addresses of memory within that pointer-memory block and then double-dereference them to get context data (and it does seem to!), your LLM's evolved logic can do "dynamic random-access reads" of arbitrary parts of the long context.
But, of course, the model won't be doing anything special with that data. It can attend to it, but not in a way that's unique to that data. It pulled in such data through generic logic that doesn't "know what it's looking at", after all.
---
IMHO, this analogy helps to make clear that while giving a model a raw "large-context capability" might help said model to attend to more data, it doesn't really help in making a model able to attend to a larger prompt.
Just by information theory, an instruction-following capability must imply that something analogous to logic keyed on those instructions gets embedded into the weights; and that logic, in order to activate and work, presumably requires its own state — i.e. requires that individual values from the context are getting attended into additional top-level named variables.
And while the short-context attendance mechanism does that, the long-context attendance mechanism inherently does not / cannot.
And this means that there is likely a practical upper limit (in terms of practical model-weight size + latent-vector size + etc) to "how much prompt" a model can follow during a single inference step. Regardless of how much data the model can "see" on a given inference step, its active "program logic" is only defined in terms of so many concurrent stateful computations.
---
Thus: trying to use a current long-context model's "low quality" attention to attend to a large prompt is just going to result in nonsense. None of these model families have ever been trained to follow a million sentences of rules simultaneously when answering a question. They're just going to attend with their inherited bounded "high quality" attention to your prompt as best as they're able, while forgetting literally everything about the prompt that doesn't fit in that bounded "high quality" window.
And anything that allows current long-context models to appear to interact well in multi-turn conversations with memory + KV caching + etc layers in operation — while also making use of its "low quality" attention to attend to large data — is some kind of hack, and will break like a hack.
Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like that's fucking sane at all. It's like trying to program by navigating a git history. Not writing any new code, you don't have the ability to do that. No, you exclusively have the ability to move around a git history and cherry pick things. It's insulting that they want to charge money for this shit.
Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.
The problem arises in the case of anthropic, openai, etc. where you can't do those things. I think they have LoRA services that they'll charge you for, but that's about the extent of it. Think of it like having a compiler and its source code, and the operating system it runs on vs simply having access to invoking it via a web service.
Look, I enjoy local LLMs as much as anyone, but I think there is some motivated reasoning happening in this thread to try to make local LLMs sound like a utopia against those evil VCs.
Local LLMs suffer from the same problems.
- They have no other options
OR
- They have no idea what they're doing
Local models solve the first problem, and let you apply the appropriate tool by virtue of the control you have. The alternative does not.
In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model, and that having this baseline of control over a computer program is "utopian". To me that sounds more like the second problem, and nothing can help you there.
No, I’m talking about the idea up thread that having control over more settings removes all LLM defects with local models.
I have all the control I want over the local models they run. No combination of settings is going to make them outperform Anthropic’s models.
> No combination of settings is going to make them outperform Anthropic’s models.
And even this is addressing something nobody was saying, and manages to come off as a marketing read.
It absolutely does not.
Like, lets say you already did the sane thing, your model[1] is already either FP16 or Q8 (and quantized by a competent practitioner of the art, ex: unsloth or bartowski), and your KV context is already FP16 or Q8... which means you now are already ahead of the major companies.
Google, OpenAI, and Anthropic heavily compress both K and V to insane levels, which might not appear to be so bad on short prompts, but especially with thinking enabled, its sort of the equivalent of JPEGing a JPEG repeatedly. Every time the model thinks, and records its thoughts into the context, and then reads it back later to think more, it becomes further and further imprecise.
All models with heavy KV context compression go off the rails somewhere between a quarter and a half of a million tokens. Every. Single. One. Every team that releases a model that has a limit of a quarter of a million did this on purpose, and it was the smart thing to do.
Now, lets say you dip your toes into samplers; this includes stuff like temp, top k, top p, min p, etc. I won't describe what they do, there are already good ELI5 articles out there to help you with that. They are, however, the original samplers, and the only ones the big companies use. None of them use the newer samplers that massively outperform them.
You know what you get with most providers? Temp as a knob, and it only goes between like 0.0 and 1.5. What if you want higher? Nope! What if you want to tune the other knobs? Usually no, too (OpenAI seems to still offer it on their higher end API plans, but Google has eliminated all knobs, and Anthropic apparently removing everything but temp in the future). What if you want other samplers? Not allowed.
Even restricting yourself to normal samplers, what if I wanted temp of 100, and min_p of 0.9 and no other samplers? That produces sane results for creative writing tasks, yet I could never do this, even I was paying for some $200/mo plan at Anthropic/OpenAI/Google/etc.
All of these samplers also cause a sort of JPEGing a JPEG repeatedly sort of error, it is the third source of it (model quant and KV cache quant are the other two). Long context insanity is probably caused more by sampling error more than it does by model and KV quant error.
What other samplers are there? Llama.cpp impls dynatemp, mirostat, top-n-sigma, and some others.
The one that I think more people need to look at is top-n-sigma. Temp and top-n-sigma alone has produced results that, even on ridiculously complex and purposefully tricky prompts, let models that have 1M native context happily go to the very limit of it without any signs of tell-tale degradation.
Want to go have fun with your new found freedom? Get Qwen 3.6 27B in Q4_K_M from a reputable dealer, set llama.cpp to do Q4 KV (yep, after I just said don't do that), and then run it with `--samplers "temperature;top_n_sigma" --top-n-sigma 1.0 --temp 1000`, and then compare it to the normal recommended values of `--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00`.
You will find that a lot of problems suddenly go away: long context degradation vanishes (which Qwen 3.6 will still do inside it's 250k context, especially when both model and KV are below Q8), forgetting what you said or it said earlier goes away, losing the plot half way through goes away, overreliance on cliches (Qwen has its own form of Claudisms, but are more subtle and less grating) also goes away, hallucinations happen far less, and lazyness also goes away (Qwen 3.6 has no real lazyness defects, but Gemma 4 does).
I have tested those top-n sigma settings on code generation as well. It is better than stock, and better than the commercial offerings by any of the American providers, but I still don't think LLMs can replace human programmers: it still can't think nor reason.
But yeah, moving to more modern samplers has done more to unfuck LLMs IMO than anything else you can do.
[1]: Assuming it isn't a native 4 bit model of some kind; quantizing them correctly to work in a local inference engine without actually quantizing anything is a bit of a PITA. See unsloth's work on the QAT Gemma 4 releases.
This is just not true. Any local LLM you can host on consumer-accessible hardware has all of these defects, too. Adjusting the knobs doesn’t solve everything.
The closest you can get to frontier performance is Kimi K3, but you’re not hosting that unless your budget is on the order of a nice house in a good metro area.
I like my local LLMs as much as the next person and my office is currently uncomfortably warm from the amount of compute happening, but I would never agree that local LLMs solve all of the common LLM defects. This is peak wishful thinking.
In my experience, the local models and even the larger ones that we can’t run at home suffer more from long context degradation than the frontier models. You are exactly right that you need to manage context length, but even at fp16/bf16 the local models have a lower ceiling for usable context length in my experience.
At the heart of it, self-hosting liberates your use cases from all the horribly opaque configuration, shadow prompting, etc. And local models are only getting better and more diverse every month.
True, almost like magic and magic are not the same thing, but I'd hesitate to call this a strawman.
that is almost exactly what they said though..?
"Want it to go away, almost like magic? Local inference. [...] all of the common LLM defects will go away.
-----
edit: I suppose I have to spell this out.
"[...]" = "When its under your control, and [you're] no longer being forced to hold it wrong," ≈ "liberates your use cases from all the horribly opaque configuration, shadow prompting, etc."
and
"Almost like magic" is cliché hyperbole. It also means "not magic" in the same way that "almost like a dog" means "not a dog." It was immediately followed by the explicit clarification that was surgically omitted above, in absolute and conscious bad faith.
Hence, "[...]"
people can just read the original from the comment. its not like im quoting from a different website and trying to hide something. look 2 comments above mine and it is right there in full. anyone reading my comment would have seen the original comment already...
i have no idea why you took such offense to my comment, but you're reading way too much into it.
Incorrect. The omitted text doesn't change the relevant meaning of the quotation in the slightest. Why are you taking such offense?
> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
> "Want it to go away, almost like magic? Local inference. [...] all of the common LLM defects will go away.
There's no difference between these two statements in the relevant context of the article, which is specifically "long policy documents do not reliably govern agents", or the comments above, which are specifically addressing how that issue is not solved by local agents.
It’s an understandable view, but I’d be astonished by any local model processing very long context better than any frontier model (and now many racks are we talking).
The parent comment literally said that the common LLM defects would go away.
Direct quote:
> all of the common LLM defects will go away
While it was not spelled explicitly, it should be obvious that nothing can be improved if you run locally an LLM in the same way as the vendor. But you should not do that. It is easy to use better methods for sampling procedures and for prompt and context handling.
"Yes, I'm using AI to speed up development. I'll submit that PR in two weeks time!"
Not only in big desktops, but even in most recent mini-PCs, it is possible to read concurrently from one PCIe 5.0 SSDs and one PCIe 4.0 SSD, at a total sustained reading throughput of around 20 GB/s.
With an optimized inference implementation, it should be possible to overlap completely the computations with streaming weights from SSDs.
This should improve the inference speed to around at least 1 token per second, on a cheap computer, under $2000 even at the current super-inflated prices.
There are enough tasks where this would be useful. Obviously one should use for most tasks a fast small LLM and use the big one only when this actually saves time.
I noticed even with plenty of context, the model sometimes chose a path that "fools" the attention layers to bypass some rules. you can notice that the rules are not ignored, they are sidestepped. it's very predictable emergent behaviour after you see it happen. with more context, it actually gets worse to the point you will see some things obfuscated even.
And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; reasoning depth is limited; and many real-world policies are not actually meant to be implemented in the same way they are written and have insufficient specification of edge cases
With humans, we usually do the equivalent of RLHF, both via "training" with simulated cases, and via feedback while on the job. You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month
The problem is that it doesn't fit the sales pitch of LLMs and agents - humanlike or better, repeatably, 24/7, for a fraction of the price, you just need to make sure that you give it all the rules.
Unfortunately we can't really have a meaningful conversation until the money vampires have left so we will need to reschedule this until after the bubble.
How many arbitrary rules can you memorize? Do you even remember them two years later? If you do, then we can get to your statement.
And your statement is true. Humans do not run every action through a memorized list of rules, to see if any of them block the action. We don't. We're not going to, either, no matter how badly the policy manual writers want us to.
without trillions of examples of following instructions at a million context length, im not convinced the behaviour is in the weights to begin with
Is it, alternatively, because loading your LoRA plus someone else's can't be done in the same batch, so inference is made harder or becomes impossible to do at scale?
Or something else?
It is about context window too so no matter how much money you put to train your model it can still have degraded performance in following instructions but it will be better to execute the instructions they can follow.
It is also about the harness and how that can help drive the model and pick and choose what to include or not in the context.
These LLMs do not understand the project. For them any next word is good as long as it was picked by the token predictor. It does not have any way to understand but only to do. probability distribution over their vocabulary and if that vocabulary is tainted and lost parts of the original context what is a good candidate there will not match the intention of the initial project.
I would imagine that doing something similar (using agent skills for insurance) would work much better for AI.
So the behavior policy should reside in the weights, not in some ever expanding KV cache context, like sticking it in the wrong orifice if the goal is to be reproductive.
Instead of putting the policy document in some tightly crammed RAM, shove it in the weights that already exist, online / post training.
It does make one wonder if there is some way to compute the change to the weights from the computed context (so that we can free it up) without essentially continued "pre" training RMAD after each communication round.
Hmmm, if AI has to grow in the workplace, it has to follow processes to the letter, yet claude code forgets by the second turn my "don't commit" prompt.
Of course claude code is a generic and crap harness over terrific models that can't fit bureaucratic processes, and is increasingly worse at doing so anyway since Opus 4.6 peak.
I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things in a prompt during the task, it performs way better.
Result is I'm trying to resist adding more and more things to CLAUDE.md files which in some scenarios it does well but in other scenarios totally ignores and messes up.
The coding harness is what's getting it to continually align to your current instructions.
This is very obvious with local models.
<system_prompt>
<tools_etc>
<project_prompts>
<user_call>
<llm_response>
=>
<system_prompt>
<tools_etc>
//Removed project_prompts
<user_call>
<llm_response>
<user_call>
<project_prompts> //reinserted
<llm_response>
Cache would break but if you did instead
<system_prompt>
<tools_etc>
<project_prompts>
<user_call>
<llm_response>
<user_call>
<project_prompts> //Duplicated with new user messsage
<llm_response>
It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message
So a good harness should be popping tool prompts, user hints after every return from the top so the context of tools and user hints never repeats and are allways prefixed to the current user prompt.
I've never inspected any harness, but opencode with the dynamic context compaction plugin appears to do this well.
Each super's sub functions should _fail hard_, and each script should be highly detailed; of course I'm not doing it myself, but in small increments of directed work, it can build up the necessary harness.
What I get is a CI that just starts with "run super-run.sh" and that gives it context, then each sub script provides context depending on if it succeeds or fails. If it fails, the agent is provided what it needs.
It's basically, you have to design the products of the AI to give itself the context. Another technique I'm testing out is a parallel set of files like <subject-module>.js, <subject-module>.test.js, <subject-module>.md which get pulled up if the Agent is looking for a file.
CI runs. Local Git hooks. Cursor also has hooks built into their agent. Other agent APIs probably have something similar.
The review subagent has its own definition and gets invoked with a specific target, so the context is very focused on just rule enforcement and I don't have problems with it skipping rules.
The whole workflow is packaged up as a plugin, but you don't need that to get this approach to work. It should be sufficient to have the rules you want enforced written out somewhere, and to either kick off a focused review agent manually referencing them, or do something like I did and have it be a defined part of your workflow (depending on the automation level you want).
In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(s) with some bullshit reason. But it is far better than assuming it will comply every time.
Then there is the "hard control", which is the inviolable that the LLM cannot bypass.
You need to move as much as is technically possible to either hard or (failing that) advisory controls.
you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.
https://github.com/anthropics/claude-code/blob/7ef6eec9d9ba8...
These seem like they would naturally extend to LLMs as well.
(I'm not pro-LLM on balance, but the solutions of today feel similar for humans and LLMs.)
That's why it's critical to have these checks run in a context that the model can't bypass, such as via github actions that block PR merges. Generally works pretty well for stuff at work (where we have all this stuff set up as part of continuous integration checks), but it means that for personal stuff I have to set up quite a bit more infrastructure to ensure that models don't just skip running tests (which they LOVE to do).
A good programming harness is basically a stack. A good stack keeps building each layer. You _cannot_ pull things off the bottom of the stack because that's an expensive cache hit; but you can pull things off the top. So what your harness should be doing: <tools> <rules> <user content> onto each request _then_, when you get to the next request or result, pulling that out if there's some change.
So you can see it's the cache that's either exponentially growing or having to cache bust to keep it fresh.
You need to let go of the idea that this is something LLM's can do. They can't. At best, they can bias towards rules conformance with more or less likelihood, but coverage and conformance both go down super-linearly as you accumulate more rules, more context, and more output in a session. That's simply the nature of how these tools work and you need to engineer your workflows around it if you want to use them.
If you absolutely need some rules enforced, you need to adopt some framework for validating those rules that then rejects, reprocesses, or repeats any session that fails to satisfy them. In the best case scenario, this is some traditional deterministic validator (like a linter, compiler, analyzer, exhaustive test suite, etc in coding) but if you need to process in stochastic space because its something rich and ambiguous like natural language itself, then you want to dispatch a swarm very narrow, task-focused subagents that each validate against a very constrained subset. (And prepare yourself to have those to fail sometimes too. LLM's are noisy and cannot deliver strict rule enforcement on their own.)
If the LLM wasn't post-trained to adhere to specific handbook, it just won't work. If the LLM wasn't trained on an use case the lab decided was worth making a synthetic agentic dataset, it won't work as well as you want.
There's a reason the main agentic task LLMs excel at are coding tasks, it's the way of working of the creators, and they understand intimately the flow and can train for it.
I believe the true way will be able to easily fine tune models on your agentic use cases, but it would require a big company to compile a huge dataset on it's way of working and I don't think anyone wants to be the first.
In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
If people spent more time studying about AI and how it works, they would realize that the default should be to one shot prompt your task with a big, cached system prmopt, with an user prompt that is just dynamic data, specified to the cheapest model that can do the job.
Unless you really can't do this given your problem, you should try to make a graph of well defined, step by step oneshot prompts, and THEN if your problem still can't be solved with that, then you start leveraging agents.
Despite this giving better results, and being more cost efficient, is evidently too much work then just letting the AI do all the work.
Geologist report -> LLM call extracts minerals we are looking for (you inject a db query result on the user prompt), locations, assay mentions and risks into structured fields -> LLM call classifies evidence into positive indicators, negative indicators and unknowns -> LLM call estimates deposit potential and confidence -> database lookups inject regional ore demand, nearby deposits, infrastructure and historical yield data -> LLM call combines geological evidence with business context -> LLM call generates an investment recommendation and rationale.
That's what I mean by graph. Every step is a separate LLM call with a well-defined responsibility, consuming the output of the previous node. Each node can be tested, benchmarked, retrained, replaced, or monitored independently. Why would you use an agent here? You can cache every single system prompt on each call making your total token output much cheaper than having a full 'output' only token generation workflow which is what happens with agents.
There is nothing to discover. The workflow is already known. The company already knows how geologists evaluate prospects. The company already knows what data sources matter. The company already knows what the final output should look like. You don't want the model deciding which tools to call, which reasoning path to take, or which pieces of information are important every single run. You want the exact same process applied to every report so results are consistent, measurable, auditable and debuggable. My default is: One-shot prompt -> if not enough -> graph of LLM calls -> if not enough -> agent. A surprising amount of enterprise AI is really just: Unstructured input -> extraction -> classification -> enrichment from databases -> decision support. Not: Unstructured input -> autonomous agent spends 20 steps deciding what to do next.
Agents make sense when the workflow itself is unknown.
If the workflow is already understood, a graph is usually cheaper, more reliable, easier to evaluate, easier to debug, and less dependent on whatever synthetic "agentic" behaviors happened to get reinforced during post-training. I am sure people default to agents mostly because it's less engineering work than explicitly modeling the process.
That's why this paper is important. They gave it a scenario very similar to what would happen if an AI-frenzied executive suddenly mandated that AI be used to do basic company tasks - a scenario that is playing out all over the place right now.
i thought it was well known that claude code got good at coding because anthropic bought tons of coding data from companies like mercor.
"Every gambler knows the secret to survivin' is knowing what to throw away, and knowing what to keep." - Kenny Rogers
Humans have limited context, just like AI. The difference is that humans - at least some of the time - can figure out which pieces are more likely to be important, and therefore prioritize keeping those in the context.
From my experience this holds true to this day. It was one of my core observations for similarity to the limitations of human working memory on "Engineering for Bounded Cognition"
If the stakes are high, e.g, proceeding in ignorance could lead to prison time, people will favor inaction, even if the policy technically permits a corner case. If the stakes are low, people will completely override policy for the path of least resistance.
I feel like “discretion” parties missing. Do we need some kind of special discretion model?
I feel this is like few-shot in reverse. Few-shot is supposed to be good examples AI should follow. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.
I wanted to know if there is any difference between writing rules in the prompt, writing them in CLAUDE.md, or not writing at all. So I did some short tests before. I asked Claude to open brand new sessions, test different topics with the rules I want to apply. The result turns out to be: in a fresh session, no matter the rule is in a prompt or in CLAUDE.md, models (Opus 4.8, 5 or Fable) all follow it fine, across models. Even Opus 4.8, the one always violates rules in our conversations, does it well.
I suspect it's the long context that breaks rules. But simulating a long conversation experiment is kind of hard, I still haven't found a good way to test it. So seeing this paper now, it completely answers the question I was stuck on these few days.
Besides, something caught my eyes in this paper: sometimes the model does run the check by the rules, and it really finds the violation, but its narrative still insists on its original wrong output.
My current approach is same as everyone here: use a separate hook or post-check to fix things. Because if you let the model fix it during generation, its narrative or main generation part sometimes just rejects the rule error it found.
In Pre-LLM days the 'nearest unblocked neighborhood' problem, where patching out one issue just immediately runs into another issue, or a different path back to the same issue. Since the models can learn new long time behaviors it's difficult to change the behavior without changing the context quite a bit.
Sure, it uses tokens slightly faster in the prompt, but I find it reduces overall token use, you can use it with pro, but of course, nothing works 100% of the time, but it's better. Emptying the memory helps too to avoid Claude making up stuff that messes with how I want it to act.
The general gist of inject_rules.py is :
RULES_PATH points at RULES.md
reads it with encoding='utf-8-sig' so the BOM is stripped
wraps it in a JSON object — hookSpecificOutput.hookEventName = "UserPromptSubmit", additionalContext = a preamble plus the full rules text
the preamble is the line you see above the rules: rules are in force for this turn, run rule 33's five tests before raising anything unasked
prints that JSON to stdout, which is how Claude Code takes it in
on OSError it returns 0 silently — if RULES.md is missing or unreadable, nothing is injected and the turn proceeds with no rules
Specifically, having agent implementation drift from the Spec.
I've done a ton of testing and the model doesn't matter, fable or sol still miss a ton of detail and drift.
I'm building http://engine.build which closes the gap and makes sure the implementation matches the spec.
It's not the same as the satisfaction you get when solving complex problems with code yourself but writing clear specs and thinking through the problem is still very satisfying to me.
I noticed that the issue tracker I built was well suited to address this problem, as it handles board time-travel. So a simple command like ":replay 4h" gives you a clear overview of how the workflow has evolved over time, and you can checkout previous states at will.
Wrote more about it here:
https://dev.to/ljtn/vision-drift-addressing-the-next-problem...
At some point between the model i was using and the previous version of it, Claude started inserting massive comments with references to tickets and other tasks. All this while having specific directives on the CLAUDE.md
Since then I resorted to developing my crapware as if I was the floor manager of a vehicle assembly line, and I have a few highly-specialized sub-agents running errands around the main session, but only ever taking care of a single concern. The main session builds with the knowledge contained in things like CLAUDE.md but the sub agents make sure things like the no/low-comments directives are either enforced, or factored into the final product.
Codex has been pushing things to my main branch all week despite me repeatedly telling it not to and adding to my AGENTS.md very clear instructions for creating feature branches and putting up a PR. It keeps doing it in spite of all that.
I'm probably going to need to enable branch protection on my personal projects... What a pain.
The solution to that is pretty simple: get to the point, list all the requirements that they must do complete for that task, constraints on what they must not do, and optional recommendations for them to follow. Leave things that they can figure out on their own out of these documents.
So, it's more about being concise.
So far I am observing two things:
1. For smaller models, performance on Benchmarks such as DeepPlanning does increase significantly.
2. Context hygiene for sub agents becomes much simpler, since that can be expressed relatively concise and the mechanics are handled by the runtime automatically.
Still looking for a good test cases to study possible advantages, but running reliable benchmarks does take time and money...
The point about multiline prompts is very valid obviously, that's on the todo list.
Multiple rounds of generating small contacts documents that grow from the original idea , trying to keep each slice small enough to process for a human to approve/disprove .
Eventually it leads to a long list of tasks grouped by functionality. You start a new context and the orchestrator agent dispatches tasks to sub agents with a limited amount of information provided to each sub agent.
Also should have adversarial review and approval gates with other agents and roles.
If I want to be sure I'm using my own deterministic rules checkers as a tool, that has to AI run and update the result accordingly.
It's hard to understand what's going on with Grok. It's like it has capabilities in a theoretical sense but maybe the training is so focused on being in x.com/grok.com with the web search tool enabled for "is this true?11" type queries that with any API type usage with document workflow instructions, tool use, code gen etc it completely falls over
https://venturebeat.com/technology/cursors-composer-2-was-se...
I prefer this behavior in a lot of situations. A plausible request can also be a genuine one.
Alignment at the grain of each tool response is way more effective than a static system prompt. I try to keep mine under 500 tokens. Why bother fighting recency bias? It's pretty much what you want most of the time.
When "performance" breaks down over long lengths, one could attribute it to a lapse in attention, but one could equally suspect inconsistent instructions.
The fewer instructions and conditions that need to be simultaneously met the easier it is to comply, but with more and more instructions one is bound to introduce internal inconsistencies within the instructions.
How many times have you told an agent not to do something then had to correct it?
You must always flip the frame. Objective analysis is way better with llms than steering via skills.
This is just a small example of why "loops" became popular for a minute and now it is "graphs"
Plus the model's capacity to take more context into account and actually integrate it to the output is simply limited by the number of activated parameters. If you give it a playbook, you are forcing to choose it between attending to the playbook and the task at hand.
If you want to force it to work step-by-step, you need to present the steps one-by-one. Ideally with rules for the current step at hand and maybe relevant input again, depending on overall task size.
Why did you think models love to re-read files before editing them? It increases recall quality and thus edit precision and thus benchmarks.
not sure I got it?
Separately, the frontier labs are kinda pushing us into that behaviour by releasing models with ever-larger context windows.
But "Read AGENTS.md, including the middle" sure helps.
Dont try and handle the entire project in context.
Use a well structured filesystem layout for your code with a few lines in an AGENTS.md describing the layout and core architectural requirements (no more than that, as per the article!).
Then work on small-medium tasks at a time with a fresh context.
At the end of your task, ask the agent if there are any key points about the project layout or architecture it would want to add to memory - audit those manually and amend your AGENTS.md accordingly.
If your code is well structured and you keep your tasks localised, you can get away with seemingly minuscule context windows.
It's also worth noting that high effort models love to slurp up whatever context they can. You almost never need/want high effort for non cross-cutting tasks.
Glad to be able to put some numbers on it.
It does what its has been trained to do. So find out what its trained to do and just use it to do that. This is not general intelligence.
Serious question: why would anyone expect otherwise?
Policy-in-English? Model implicitly complains that it's TL-DR.
Ask the model to write code that checks your policy, then add that code behind a simple validation hook (e.g. "check your work by running 'just validate'") that the harness knows to always run after changes? It suddenly becomes the most law-abiding citizen ever.
The prompts reflect the actual jobs enterprise workers perform every day. Each task drops an AI agent into a live company environment, requiring them to cross-reference an extensive, multi-section handbook against a cluttered inbox, a multi-channel Slack workspace, Jira queues, and a stack of files (spreadsheets, PDFs), and working out both what to do and what the handbook forbids.