What are the choice cuts?
In an HBM bandwidth constrained setting you're dealing with something called "roofline analysis" (comes originally from NUMA work circa ~2009 but it's applicable to modern GPUs). Great diagram from the JAX people:
https://jax-ml.github.io/scaling-book/roofline/
In order to get your money's worth from a modern GPU (or disagg rack like an NVL72) you need to decode (the one token at a time thing) across big batches of context windows. To the left of that point where it hits "the roof" you're idling tensor units. TensorRT-LLM likes batches of 4096, so BS=4096.
In the case of one person chat prompting their local LLM, BS=1, totally bandwidth limited.
So the game is to set some latency target with some control theory primitive (PID or something) and then delay the next token until a batch is big enough to not waste tensor units. This is a real trick when a human is waiting (you've probably seen the thing in Claude.ai where it's all bursty and then they reflow the whole block with JavaScript).
Agentic workloads are huge piles of context windows where you've always got enough who want the same experts on the next token, you're always to the right of that intersection. And it doesn't really matter if it's on the other side of the world, or lags by a second, it's fine.
Claude Code soaks up all the tensor units that would be idle until they're full, and only then does it leak into the capacity reserved for highly interactive use. It's the bottom of the barrel until it's rinsed the fuck out.
They want more margin on agentic tokens. That's it. The COGS on them is the absolute lowest of anything they do.