Oh neat! I'm a little slammed the next couple of weeks with an existing engagement + CAIS, but happy to connect and see if timelines work out?
328 karma · joined April 7, 2026
Context compaction can also affect the outcome - I have eval scenarios for that as well but not in the published set, only in the repo. For those, I'd say "it's better than nothing". If you hit max context, the whole thing will barf or OOM the rig or something like that. So compaction degrades performance versus some theoretical ideal where you never need to, certainly. But it's better than a hard failure. Eval on those scenarios showed increasing degradation depending on severity of compaction. I view the auto-compaction as insurance. I never give the models tasks that will require that much context, but if it ends up getting there then the run might be saved.
But at the end of the day, if the model keeps responding with text, there's nothing forge can do. I've run into that failure mode for sure, even with forge.
That works well enough for all the models shown in the eval here: relatively modern 8B+ models.
But some of the older generation (mistral 7b, that sort of thing) still can't be reliably used in something like a production setting.
I do think there's some differences though. The biggest one being that forge isn't a coding harness, it's a guardrail primitive, really. Applicable to any tool-calling workflow.
As for the errors, are you nudging or passing errors back or swallowing them completely? Love the 2-stage routing though, neat!
What project are you referring to specifically?
ToolResolutionError is really inspired by HTTP 4xx vs 5xx codes. I don't even have a super clean abstraction I'm happy with yet, I just noticed a lack of standard in the industry (that I was aware of) so I thought to surface it as a gap. I'm sure there's a better shape than my current ToolResolutionError but it's a start!
Harnesses do have retry mechanisms. In opencode in particular, I think they return the error as-is to the model in the next turn. But that's slightly different. Harness retries come mostly in two flavors:
1) provider-layer: HTTP requests to cloud retries, with or without exponential backoff. It covers you for transient network hiccups or rate limits, and a big Opus model really doesn't need more than that.
2) sort of a hope-and-pray retry. Tool ran, returned an error string of some kind, gets fed into model as-is, and the model is expected to read the error message and self-correct with no guidance. This is fine for frontier, and even some of the large oss models. They have the context-following capabilities needed. For smaller models, this won't be enough, not reliably over many turns.
- if model outputs malformed json, provider will reject it before it even reaches the tool, the error loop is broken. A rescue parser handles that - can be ~5-15% of calls on a small model sometimes.
- model calls the wrong tool, correctly, then proceeds confidently with context that won't help it. step enforcement can help here.
- model terminates prematurely, thinking it's done. prerequisite enforcement can help here (say, forcing the model to call pytest before declaring the feature built).
- Escalating nudge messages, that specifically nudge. Just returning error messages doesn't tell the model what to do, it just tells it it was wrong. A message that spells out "tool X does not exist, call one of the available tools: A, B, C" is more helpful to a small model than "error: X not found".
So, in short - yes, retries exist in harnesses, but rely on top-tier model interpretation of the error messages. When working with top models, there's likely no real difference, or a minor one (see Opus bare vs Opus reforged). But Forge provides a more hardened suite of guardrails that are effectively necessary for small models.
Mostly, I'm embarrassed I've done this whole public reveal without any use of Alanis Morissette anywhere in the work :/
But otherwise, forge really doesn't own or opine much of the workflow. Step enforcement exists if you want it, so do prerequisites, but the idea is that those could be conditional or optional (you may never need to edit a file).
The guardrails are designed to work for non deterministic flows or deterministic ones. In the latter, you just might not have one of the guardrails active. It's much more about nudging the model back on track than laying more obvious tracks, in a sense.
Overall, agentic reliability is definitely an active field.
I think this sits a level higher than Forge - maybe testing the workflow proper and integration points that it might surface (if some tools are giving access to an MCP or something).
Could likely layer both together without much trouble.
Only thing I'd be curious about is how you handle the non-deterministic nature of these models. Sometimes they get the tool call right, sometimes they barf bad json. Does the suite run multiple trials?
Proxy mode should work fine with remote models, the only constraint is the compatible endpoint - which is standard anyways. I don't think you'd have any issue hitting either a remote gateway like liteLLM or just claude API.
Small models aren't there yet and they would veer off course, this just nudges them back onto the road. Whether or not they have a good sense of direction is a different question.
The isssue/use-case is more around, say, a database table or legacy systems where your tool is just hitting a legacy API that may or may not be good. A surface you don't control.
It didn't come up as a use-case in this eval honestly, it's more the concept of a standard, like 4xx vs 5xx. I just felt it was missing from the ecosystem overall.
Did you notice any particular guardrails firing? Always curious about things I haven't tested on - especially if it has a different shape.
Plus it's cool to see a little 8B model writing code :)
Without forge, I'd guess a small model used for Hermes would have to retry entire workflows when an uncaught exception triggerd when it tried to reply with text when "calling a tool" ("Here is the tool call: [json blob]"). The issue there becomes partial successes can lead to state changes that need to be addressed (it booked the flight already, home it doesn't double-book).
Forge won't help with model reasoning quality though. If it the model thinks the right thing to do is to book 3 buses for your trip, forge doesn't care, it'll just make sure those api calls land.
For the paper - more academic in nature - I wanted to isolate the model performance variable from guardrail lift. The delta is what mattered more than final score. For the paper, everyone got temp=0.7 - that was intentional.
As for Qwen3.6, it's really solid. It'll do really well on forge I can call that now. When I pushed it into agentic coding specifically and the eval suite I use there (separate from forge), even it needed help on long-running tasks - but it's definitely a top model right now.
However, entirely possible there are better settings than the "official recommendations" I found - which would be a neat finding in itself.
In a nutshell, it applies guardrails around LLM calls to make them more reliable - specifically small models but works on all: "on multi-step agentic workflows through guardrails (rescue parsing, retry nudges, step enforcement) and context management (VRAM-aware budgets, tiered compaction).".
It'll try to parse malformed tool calls, it'll automatically compact if needed, it'll enforce any workflow requirements you define (ie, read before edit) - and it does so with domain-agnostic guardrails. It catches and feeds errors back to the model in a structured way so the model self-corrects (hopefully).
Each guardrail can be removed as desired by a consumer. It can be used as a building block library (WorkflowRunner approach), it can be integrated into existing source (middleware), or it can be a drop-in addition to an exiting workflow (proxy mode).
I feel like there's some backend decoding or chat template thing going on at a much lower level than what I'm best at. Maybe it's injecting headers or something that eventually compounds to model confusion? I really have no idea.
I really hope folks better than me at backend stuff take a look and dive into it though because it's definitely under-reported and super consistent across model families and backends ranging from ollama, lama.cpp native, prompt, llamafile, and even vLLM that I didn't formally benchmark in the repo.
And if you didn't mean that then please elaborate :)
A few things I noticed related to your points: - on conversation rewind, I implemented a similar tool call collapse on the main agent (the one you chat with). Once it was done with a task, the tool call history was collapsed to keep the context clean - it was more about hygiene than size.
- the harness interrogating the model bit is a bit different, I haven't tried that approach. Forge relies on model self-correction in a bid to avoid having bespoke error modes, but I guess if you can abstract and automate the interrogation based on schema or something that could work!
Overall I like the clean conversation history aspect, but I suspect that you might be doing a lot of round trips for tools with many args, versus "letting it fail and giving it one nudge". That being said, it's an interesting idea for harder scenarios/tasks!
You're 100% right about how I meant it and what it means within Forge though, but it's something that might lead to doc changes as things evolve.
I'd assume they could be combined. A coding harness would own the agentic workflow by nature, forge guardrails would help tool calling.
I haven't given it a thorough read yet but I think their guardrails might be more focused on the workflow level. They are doing error capture at tool level with warnings to the model, but I'd need to dig deeper. On the surface definitely the same design philosophy! Maybe Forge makes error nudges more of a first-class citizen?
Our compaction strategies might be the most similar of all the pieces. Cool find!