__jl__

artificialanalysis.ai benchmarks are [here](https://artificialanalysis.ai/articles/claude-sonnet-5-5). Anthropic is back at spot 1, 2, 3 and 5. Impressive even if these benchmarks are problematic in many ways.

This is really a great achievement: "Astra dominates the output token frontier"

Many labs used increased thinking to boost benchmark scores and performance. Most of the Chinese models were doing that for a while. Google and Anthropic as well.

Not OpenAI. 5.6 already was much more token efficient than other models and Astra beats Sol in token efficiency by a wide margin.

Edit: Just to make the point: Astra (max) has the 2nd highest score and the third lowest output tokens (among the models shown by AA).

Didn't expect that. Luna pricing is crazy now. I don't think there is anything on the market that competes at this price-performance point.

For our production app, OpenAI clearly is the best provider now. Their API is very reliable and has many nice features. The price-performance of the model lineup is incredible. We used open weights model via Fireworks for a long time (e.g. Kimi K2.5). Fireworks is a great provider but we still ran into issues here and there (Same with Anthropic and Google). OpenAI just works, is fast and in my view has a better price-performance ratio across almost all levels of intelligence.

This understates the cost increase. 3.5 Flash also uses more tokens. artificialanalysis.ai shows these difference to run the whole eval, which I think is more realistic pricing:

Gemini 2.5 flash (27 score): $172 (1.0x)

Gemini 2.5 pro (35 score): $649 (3.8x)

Gemini 3.0 Flash (46 score): $278 (1.6x)

Gemini 3.5 Flash (55 score): $1,552 (9.0x or 2.4x compared to 2.5 pro)

This is a massive price increase... 5.6x compared to Gemini 3.0 Flash

In our experience, caching is not very reliable with google. We always get random cache misses that don't happen with other providers. We find OpenAI, Anthropic and Fireworks (which we use a lot) all have higher cache hit rates. So it's not only about the costs of cached token but also what kind of cached hit rate you get.

I see your point. I do find Anthropic's approach more clean though particularly when you add in mini and nano. That makes 5 models priced differently. Some share the same core name, others don't: gpt 5 nano, gpt 5 mini, gpt 5.1, gpt 5.2, gpt 5.4. And we are not even talking about thinking budget.

But generally: These are not consumer facing products and I agree that someone who uses the API should be able to figure out the price point of different models.

What a model mess!

OpenAI now has three price points: GPT 5.1, GPT 5.2 and now GPT 5.4. There version numbers jump across different model lines with codex at 5.3, what they now call instant also at 5.3.

Anthropic are really the only ones who managed to get this under control: Three models, priced at three different levels. New models are immediately available everywhere.

Google essentially only has Preview models! The last GA is 2.5. As a developer, I can either use an outdated model or have zero insurances that the model doesn't get discontinued within weeks.

Another preview release. Does that mean the recommended model by Google for production is 2.5 Flash and Pro? Not talking about what people are actually doing but the google recommendation. Kind of crazy if that is the case

Impressive jump for GPT-5.3-codex and crazy to see two top coding models come out on the same day...

Yes you can and I really like it as a feature. But it ties you to OpenAI…

I will have to try that. Cursor bill got pretty high with Opus 4.5. Never considered opus before the 4.5 price drop but now it's hard to change... :)

Mostly at the time of release except for 1.5 Flash which got a price drop in Aug 2024.

Google has been discontinuing older models after several months of transition period so I would expect the same for the 2.5 models. But that process only starts when the release version of 3 models is out (pro and flash are in preview right now).

This is awesome. No preview release either, which is great to production.

They are pushing the prices higher with each release though: API pricing is up to $0.5/M for input and $3/M for output

For comparison:

Gemini 3.0 Flash: $0.50/M for input and $3.00/M for output

Gemini 2.5 Flash: $0.30/M for input and $2.50/M for output

Gemini 2.0 Flash: $0.15/M for input and $0.60/M for output

Gemini 1.5 Flash: $0.075/M for input and $0.30/M for output (after price drop)

Gemini 3.0 Pro: $2.00/M for input and $12/M for output

Gemini 2.5 Pro: $1.25/M for input and $10/M for output

Gemini 1.5 Pro: $1.25/M for input and $5/M for output

I think image input pricing went up even more.

Correction: It is a preview model...

API pricing is up to $2/M for input and $12/M for output

For comparison: Gemini 2.5 Pro was $1.25/M for input and $10/M for output Gemini 1.5 Pro was $1.25/M for input and $5/M for output

API pricing is up to $2/M for input and $12/M for output

For comparison: Gemini 2.5 Pro was $1.25/M for input and $10/M for output Gemini 1.5 Pro was $1.25/M for input and $5/M for output

Same here. They have been aggressively increasing prices with each iteration (maybe because they started so low). Still hope that is not the case this time. GPT 5.1 is priced pretty aggressively so maybe that is an incentive to keep the current gemini API prices.

The prompt caching change is awesome for any agent. Claude is far behind with increased costs for caching and manual caching checkpoints. Certainly depends on your application but prompt caching is also ignored in a lot of cost comparisons.

Since we have cursor people joining, let me bring up my constant problems around applying code changes. For background, I mostly work with "chat":

1. The apply button does not appear. This used to be mostly a problem with Gemini 2.5 Pro and GPT-5 but now sometimes happens with all models. Very annoying because I have to apply manually

2. Cursor doesn't recognize which file to apply changes to and just uses the currently open file. Also very annoying and impossible to change the file to which I want to apply changes after they were applied to one file.

1.5 -> 2.0 was a price increase as well (double, I think, and something like 4x for image input)

Now 2.0 -> 2.5 is another hefty price increase.

Same! :)

Voyage models are great in my experience and I am planing to test 3.5. Almost more interested in 3.5-lite though. Great price.

My concern: voyage api has been unreliable. They were bought by mango db, which makes me a little uneasy.

Gemini embeddings look like a great model but it’s in preview and there haven’t been any updates for a while (including at io). Also not sure how committed Google is to embeddings models.

Thanks. I looked a couple minutes ago and couldn't see it. For anyone curious, pricing remains the same as previous Anthropic models.

Anyone found information on API pricing?

Here are my two cents on cursors versus windsurf approach:

CURSOR shifted to a more agentic approach even for chat requests to reduce input tokens.

Previously, they used the good old RAG pattern with code dumps: Request with user added files -> Retrieval (when Codebase enabled) -> LLM requests with combined context from user and retrieval.

Now they seem to be doing something like this: Request -> LLM with tools to search code base and/or user-added files

I get constant search tool calls even for user-added files. Big reduction in input token but I think performance suffers as well.

WINDSURF is still willing to dump code into the context, which gives them an edge in some cases (presumably at a cost of input tokens).

Windsurf is willing to spent to acquire customers (lower subscription cost, higher expenses for llm calls). Cursor has a huge customer base and is working on making it sustainable by a) reducing costs (see above) and b) increasing revenue (e.g. "Pro" requests for 0.05 with more input and output token).

Only problem is that the genai API at https://ai.google.dev is far less reliable and can be problematic for production use cases. Right around the time Gemini 2.0 launched, it was done for days on end without any communication. They are putting a lot of effort into improving it but it's much less reliable than openai, which matters for production. They can also reject your request based on overall system load (not your individual limits), which is very unpredictable. They advertise 2000 requests per minute. When I tried several weeks ago, I couldn't even get 500 per minute.

I think the 9 billion is very misleading. More than half goes to hospitals affiliated with Harvard. I am not sure but I don't think they get anything from the endowment. The impact of loosing this money would be very uneven across different parts of the university and hospitals affiliated with it.

The faculty of arts and science would be fine. Yes, some cuts, a hiring freeze etc. The med school and public health school would feel a big impact. They employ so many people on "soft money" through grants including many faculty members.

The hospitals are a different story and I am not sure why they are even lumped together.

How does this compare with pg_search (formally pg_bm25) from ParadeDB?

They commit to supporting the API in step 1 but it's not entirely clear to me whether that commitment continues with step 2-3...

The numbers in the blog post seem VERY inaccurate.

Quick calculation: Input pricing: Image input in 2.0 Flash is $0.0001935. Let's ignore the prompt. Output pricing: Let's assume 500 token per page, which is $0.0003

Cost per page: $0.0004935

That means 2,026 pages per dollar. Not 6,000!

Might still be cheaper than many solutions but I don't see where these numbers are coming from.

By the way, image input is much more expensive in Gemini 2.0 even for 2.0 Flash Lite.

Edit: The post says batch pricing, which would be 4k pages based on my calculation. Using batch pricing is pretty different though. Great if feasible but not practical in many contexts.

How do we know that they didn't see it? Their work is much more secret now. Isn't it possible that o1 and o3 rely on something similar maybe with some additions. Same for the gemini thinking models.

My point it that OpenAI and google might have been working with very similar approaches for months.