Hacker News Reader

Comment by AJ007 | Hacker News Reader

It is only a price drop if price * tokens used is less

Happened to be testing a "review patches on a mailing list" harness I was developing; here are a sample of the latest results, testing 12 patches containing a total of 14 issues:

Opus 5.5: Found 8/14 issues. Total cost: $15.40

Fable 5.1: Found 7/14 issues. Total cost: $66.34

Opus 5: Found 6/14 issues. Total cost: $15.19

Sonnet 5: Found 2/14 issues. Total cost: $19.15

This is a relatively small sample size, but it was both the best and the cheapest.

ETA: NB this is "Equivalent API" cost as reported by claude's CLI; I was using my subscription.

I doubt your test if you cant even notice that it is not the cheapest with just 4 numbers to compare.

Heh I’ve been doing almost the same - back testing against PR comments - and opus 5.5 matched fable 5.1. About $24 for 10 PRs.

I was surprised how much worse Astra did on correctness; I stopped testing with it. Gonna try sol and Luna but low confidence

I just told Opus 5.5 "Perform a code review on the current branch" to see what it would come up with. The results were not inspiring. It told me there were five issues, one of which was a test-coverage gap on line 848 of ProjectTemplateTests.cs. But ProjectTemplateTests.cs is only 160 lines long.

I told it that it had made a mistake in the line number, and to double-check all the line numbers. It responded "You were right to push on this: four of the five line numbers were wrong, and while checking them I found two findings that were overstated."

Then I noticed in the corner of the Claude CLI UI that it was showing "Effort: medium". I'm pretty sure I had set it to high effort before; I don't know when it reverted to medium, but that's another thing that doesn't exactly fill me with confidence.

I'll try again on high effort to see if it does better, but so far I am not impressed with Opus 5.5 on my first day of using it.

My prompts are moving in the other direction as sashiko [1], a managed pipeline developed for the Linux Kernel mailing list like a year ago. But last year's models needed a lot more structure and guidance; the results I posted are from the "single prompt" version of the same thing. The README [2] describes the difference. You can browse the contents to get an idea; basically all the prompts were actually written and iterated by Fable (and now Opus 5.5), seeing how agents failed the tests and improving them.

[1] https://github.com/sashiko-dev/sashiko

[2] https://gitlab.com/xen-project/people/gdunlap/xen-review-pro...

Good data and goes to show that Fable is melting the GPUs and is priced accordingly. I'd guess that cost to serve for Opus 5.5 is meaningfully lower through architecture advances

UPDATE: Sorry, just noticed I typed in the Opus 5 total cost wrong -- it should be $58.19. Main point "best and cheapest" was from the actual numbers, not my typo.

They're claiming a drop in token use too, and that it nets to 40% cheaper.

Unfortunately, they're full of it https://artificialanalysis.ai/models/claude-opus-5-5#token-u...

It does work out to be a similar cost per task though

You should probably look at the cost/score graph by effort level instead:

https://artificialanalysis.ai/models/claude-opus-5-5#intelli...

It is most of the pareto frontier.

Not disputing the increase in quality, just stating that non-cherry-picked benchmarks show it is more verbose at Max effort

Is verboseness the only measure of token efficiency towards overall task completion?

so don't use it at max? The benchmarks suggest that high/xhigh are more than sufficient to be ahead and a whole magnitude below max with regards to token usage. I'd treat that as an outlier and not how verbose the model is in general (QED I know)

You’re missing my point. I’m saying anthropic are exaggerating their results.

how are they exaggerating the results? Comparing the cost from that chart for 5 and 5.5 for medium-max effort paints a pretty clear picture:

         mean  median
 model
 5      4.135   4.245
 5.5    3.150   2.640

Again seeing how max is a clear outlier, the median cost saving is ~38%, not that far off from the proclaimed 40%.

I don't think so, I typically use Opus 5 on High, and 5.5 scores lower on token use:

https://artificialanalysis.ai/models/claude-opus-5-5?models=...

5.5 is higher for max effort, slightly higher for xhigh and lower for high, medium and low effort.

The biggest proportional difference seems to be at max (5.5 is 38% more) and at high (5.5 is 21% less).

I think most people run at high and xhigh. At xhigh it is close enough to be task dependent and I don't think most people will notice. At high effort I think it looks like it will be an improvement for most people.

5.5 Max should probably be compared to Fable - it performs a lot better than 5 Max.

https://artificialanalysis.ai/models/claude-opus-5-5?models=...

parent means that they could get more client / a larger part of the market, which would lead to more income (more tokens) despite lower marginal prices

I tested it with Claude Code, and I can confirm it's way cheaper, better, faster and less verbose than Opus 5.

Reply on news.ycombinator.com