Towards a science of scaling agent systems: When and why agent systems work
research.google
research.google
The paper sounds too shallow. The errors data doesn't seem to have a rationale or correlation against the architecture. Specifically, what makes the SAS architecture to have lowest error rates while the similar architecture with independent agents having highest error rates? The conclusion doesn't seem well-grounded with reasoning.
I can believe SAS works great until the context has errors which were corrected - there seems to be a leakage between past mistakes and new ones, if you leave them all in one context window.
My team wrote a similar paper[1] last month, but we found the orchestrator is not the core component, but a specialized evaluator for each action to match the result, goal and methods at the end of execution to report back to the orchestrator on goal adherence.
The effect is sort of like a perpetual evals loop, which lets us improve the product every week but agent by agent without the Snowflake agent picking up the Bigquery tools etc.
We started building this Nov 2024, so the paper is more of a description of what worked for us (see Section 3).
Also specific models are great at some tasks, but not always good at others.
My general finding is that Google models do document extraction best, Claude does code well and OpenAI does task management in somewhat sycophantic fashion.
Multi-agents was originally supposed to let us put together a "best of all models" world, but it works at error correcting if I have Claude write code and GPT 5 check the results instead of everything going into one context.
> As tasks require more tools (e.g., a coding agent with access to 16+ tools), the "tax" of coordinating multiple agents increases disproportionately.
This aligns well the principle of highly cohesive, loosely coupled design for software components. If you instruct the AI to design this way, it should result in components that're simpler to reason about, and require fewer sequential steps to work on. You can think of cohesion in many different ways, but one is common functions, and another is tool/library dependency.
Even in the case of a single agent, the compounding of errors [1] can easily make your "flow" unacceptable for your use case. The deterministic where possibe/decoupled/well tested approach is key.
With such a fast moving space I'm always wary of adopting optimization techniques that I can't easily prove and pivot from (which means measuring/evals are necessary).
Slowly but surely, abstractions allow us to use others' deep investments in the matter of coordination without losing control (e.g. pyspark worker/driver coordination) and we can invest on friction removal and direct value generation in our domains (e.g. banking/retail/legal payments, etc)
- [1] https://alexhans.github.io/posts/series/evals/error-compound...
Now if I ask “why did you not use the skill”, the answer typically starts with an apology, the insight which skill it should have used, and it will proceed using the skill.
In contrast, asking “what can we change in the skill description so you use it next time I ask you to commit”, it will typically explain how it selects skills and how to modify the skill in question’s description so it would pick it on its own.
A single-agent system (SAS) uses this budget for a deep, unified reasoning stream (averaging 7.2 turns), multi-agent teams would fragment the same budget into dozens of coordination messages
I wonder if the budget is increased (say 50k) would the same results be observed ?
Good for promo projects though, lol
Empirically, a top level orchestrator that calls out to a planning committee, then generates a task-dag from the plan which gets orchestrated in parallel where possible is the thing I've seen put in the best results in various heterogeneous environments. As models evolve, crosstalk may become less of a liability.
Depth first, slow turtle recursion is likely the best way to reason through the hardest problems. It's also much more efficient compared to things that look more like breadth first search (gas town).
> Average performance (%) across four agentic benchmarks improves consistently with increasing model Intelligence Index.
> Centralized and hybrid coordination generally yield superior scaling efficiency, suggesting that collaborative agentic structures amplify capability gains more effectively than individual scaling alone.
Then again, the deltas between SAS and best performing MAS approach are ~8%, so I can't help wonder if it's worth the extra cost, at least for the generation of models that was studied.
The error amplification numbers are wild! 17x for independent agents vs 4x with some central coordination. Clink provides users (and more importantly their agents) the primitives to choose their own pattern.
The most relevant features are...
- work queues with claim/release for parallelizable tasks - checkpoint dependencies when things need to be sequential - consensus voting as a gate before anything critical happens
The part about tool count increasing coordination overhead is interesting too. I've been considering exposing just a single tool to address this, but I wonder how this plays out as people start stacking more MCP servers together. It feels like we're all still learning what works here. The docs are at https://docs.clink.voxos.ai if anyone wants to poke around!
> The part about tool count increasing coordination overhead is interesting too. I've been considering exposing just a single tool to address this, but I wonder how this plays out as people start stacking more MCP servers together.
It works really well. Whatever knowledge LLMs absorb about CLI commands seems to transfer to MCP use so a single tool with commands/subcommands works very well. It’s the pattern I default to when I’m forced to use an MCP server instead of providing a CLI tool (like when the MCP server needs to be in-memory with the host process).
That's great to hear. It makes sense given the MCP server in this case is mainly just a proxy for API calls. One thing I wonder is at what point do you decide your single tool description packs in too much context? Do you introduce a tool for each category of subcommands?
My hunch is that we'll see a number of workflows that will benefit from this type of distributed system. Namely, ones that involve agents having to collaborate across timezones and interact with humans from different departments at large organizations.
Is this going to be released for general use?
The rest is trash they are forcing down our throats
That's sarcasm
---
Your "direct Gemini calls" is maybe the least impressive
edit: This paper is mostly a sort of "quantitative survey". Nothing to get too excited about requiring a grain of salt
The products they build, where the agentic stuff is, is what I find unimpressive. The quality is low, the UX is bad, they are forced into every product. Two notable examples, search in GCloud, gemini-cli, antigravity (not theirs technically, $2B whitelabel deal with windsurf iirc)
So yes, I see it as perfectly acceptable to be more skeptical of Google's take on agentic systems when I find their real world applications lackluster
The antigravity experiment yes was via windsurf - probably nobody expected that to take off but maybe was work that made have surfaced some lessons worth learning from.
There is no clear vision, coherence, or confidence that the products will be around in a another year
Their poor product decisions have driven me away, that doesn't mean I'm still very impressed with everything under that. I'm building my custom agent on their open source Agent Development Kit and the Gemini family.
They are really cooking with Flash + Antigravity.
The Claude family seems to be better at localized coding tasks
I'm a big fan of Claude Code based prompts with Gemini 3 Flash in my coding agent. I'm unwilling to use any new Google products at this point in time. Used to be a stan, they have pushed me away and I'll never be a stan again