Here are some early rough numbers from our own internal usage on the Amp team (avg cost $ per thread):
- Sonnet 4.5: $1.83
- Opus 4.5: $1.30 (earlier checkpoint last week was $1.55)
- Gemini 3 Pro: $1.21
Cost per token is not the right way to look at this. A bit more intelligence means mistakes (and wasted tokens) avoided.
Much better to look at cost per task - and good to see some benchmarks reporting this now.
Had to modify them a bit, mostly taking out the parts I didn’t want them doing instead of me. Sometimes they produced good results but mostly I found that they did just as well as the main agent while being way more verbose. A task to do a big hunt or to add a backend and frontend feature using two agents at once could result in 6-8 sizable Markdown documents.
Typically I find that just adding “act as a Senior Python engineer with experience in asyncio” or some such to be nearly as good.
If you delegate that work to a sub-agent, it does all the heavy lifting, then passes the results to the main agent. The sub-agent's context is used for all the work, not the main agent's.
If a cheaper model hallucinates halfway through a multi-step agent workflow, I burn more tokens on verification and error correction loops than if I just used the smart model upfront. 'Cost per successful task' is the only metric that matters in production.
ArtificialAnalysis has a "intelligence per token" metric on which all of Anthropic's models are outliers.
For some reason, they need way less output tokens than everyone else's models to pass the benchmarks.
(There are of course many issues with benchmarks, but I thought that was really interesting.)
If you use very long threads and treat it as a long-and-winding conversation, you will get worse results and pay a lot more.