Hacker News Reader

We really need to stop using $/M tokens as the pricing benchmark. I've found that the number of tokens used tends to be a bigger factor than the listed per token price. The cost per task vs. intelligence curve is really what you care about, and in my estimation Chinese models are just not there. They are focused on benchmaxing and getting the highest raw score they can, rather than efficiency.

The artificialanalysis cost per task chart has DeepSeek as the clear winner and Fable as the clear loser. But I would still pick Fable for some tasks, so that also can't be all there is to it.

But I agree that price per token figure is not great. It seems even the tokens per character can vary between models, so it's basically useless.

Wow, you weren't kidding. I looked at their chart, and the cost-per-task for Fable is more than double Sol's. And DeepSeek absolutely stomps. Four cents per-task vs Sol's $1 and Fable's $3.

I might need to check out DeepSeek more. I had no idea the difference was this obscene. Makes me wonder if something's off with the benchmark. A 70x cost reduction vs. Fable seems too good to be true.

In my benchmarks of security auditing abilities of models, DeepSeek was roughly an order of magnitude cheaper than either GPT 5.5 or Opus 4.8, less than ten cents per task vs. roughly a buck each for the best American models at the time.

GPT 5.5 Pro was ~230x at almost $23 per task.

DeepSeek is my go-to when I need an API, and local Gemma 4 won't do because it's either too slow or not capable enough. DeepSeek isn't at the frontier but it's good enough for a lot of things, very cheap, and quite fast. Flash is even faster and cheaper, and still better than anything I can host locally.

Almost as if centuries of passing the imperial exam by either skill or cheating influenced Chinese culture a lot.

Yes, this is already accounted for in many benchmarks, but without deep context of the problem type, the top line pricing is the best starting point.

In my own experience, Fable is more token efficient than opus 4.8 with a higher likelihood of completing tasks correctly or at least with minimal corrective work. Opus regularly struggled to gather the correct context and reason effectively about what it had gathered.

GPT-5.6-sol crushes fable in speed and token efficiency and is clearly superior across many tasks that matter for me.

I also find all models from anthropic after opus 4.6 to suffer from the same ai slop language that long plagued OpenAI and seems to have been reduced drastically in 5.6

Don't forget that you are not really seeing the thinking tokens used - so non-trivial to count them.

Ockbench: https://ockbench.github.io/ show token usage and accuracy, but you have to compare it with other stats (prices and speed).

While I do agree that cost per task is what customers should care about, and not cost per token. Cost per token is an objective metric. Cost to do a task can vary a lot. Different tasks, different prompts, or just pure randomness nature of models make it a bit harder to define this as an objective metric.

Unless the cost per token is prohobitedly high, people can often try the model out themselves and make a subjective judgement of how effective and efficient is it at solving tasks they usually deal with, using their setup.

In my experience, Kimi just tends to think a lot, with the main thing that takes up a lot of space is it constantly second-guessing itself. I've watched it do paragraph after paragraph of "Wait, actually..." while it stumbled and used a ton of tokens on one small detail of what it was asked to do. Though I also gave GLM 5.2 a task to port some JS code to Python to test it, and in my experience it doesn't second guess as bad as Kimi does, but it really did there. It kept doing web searches and second guessing tons of tiny little things, using $0.25 of API spend in total to port about ~50 lines of JavaScript. It did produce an error the first run, but on second run it gave me a program that ran.

I gave Claude Code/Fable the same task and it took significantly less time, but also stumbled on the same error as GLM. I didn't have it fix it though. I was mostly interested in timing differences.

I do like open models where I can, but I'm really hoping they get trained to second guess less. Or maybe I just need to prompt them differently. I'm not sure.

Interesting. Could you not tell it to not second-guess itself (on minor issues)?

Fighting against the weights is often a lost cause. For any model.

Yeah I've noted this behavior with best in class open weight models. They said K3 would have token efficiency improvements and I was hoping especially solving the thinking loop issue that plagued K2.x but even if this release helped somewhat, it looks like we still have a long way to go here... I'm not sure what's up here but I suppose lacking finetuning quality.

What OpenAI in particular have done with reasoning efficiency in the past few months since ChatGPT 5.5 is nothing short of remarkable. It's overshadowed a bit by the benchmark game and the Fable hoopla.

Now is the time to focus less on token cost and intelligence, but tokens to solve a particular set of tasks in closed benchmarks for a variety of categories.

What is the use of grand intelligence if it either costs you a kidney or can't complete at all within a token budget? Even if there are niche uses where you truly want "maximum power" above all, we need to at least more severely penalize such models versus those that does it just as fine within a tenth of the token cost.

I'm aware of some benchmarks at the Artificial Intelligence site, but CLEARLY we are not focusing enough on these today and still leaving the fun surprises to the users.

Yeah, I'm finding I end up switching to Codex and GPT 5.6 a lot lately because I've either run out of Fable usage or Fable refused to do the task. Most recently it refused to work on a WiFi configuration UI for a robot. No idea why it thought that was related to security, biology, or some other sensitive topic. They've hobbled it with guardrails that are overzealous and now there's a big opening in the market. Fable may be the best, but if it won't do the job half the time, it stops being my go to model as I don't want to waste time only to find it refuses halfway through.

Earlier today I made Claude code implement a feature with fable. It worked roughly 60 minutes and used around 30% of my 100€ subs 5h sessions.

Then I typed /code-review in a second terminal/clean session after the analysis was done (no code changes) the usage was 99%. I then asked it to write that into a review.md so I could restart from that the next day. Sadly the last % wasn't enough for that.

Ymmv, these models behave very differently with no discernable reason. Usually reviews(even with fable) take like 10-20%... Yet suddenly you get it to burn through 65-69% in 15 minutes or so

/code-review in Claude Code spawns a lot of sub-agents (counted like 8 once), each looking at the code from some certain aspect (like correctness, maintainability, duplication, testing, etc). It eats tokens like crazy doing that, but also covers quite a lot. The default code review in Codex does far less (feels like it's only correctness) and doesn't uses subagents. Actually I made a skill for Codex that does a review closer to what Claude does by default, but using like 4-5 agents and some being cheaper models/less than xhigh reasoning. I'm getting pretty nice reviews with that that cover more than just correctness.

You can do `/code-review low` to more closely match Codex's default behavior.

Read the session file of the reviewer tomorrow. Sonnet should suffice.

> Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare

AI subscription pricing is so goofy. You get some amount of usage that varies by models, is measured by opaque token usage, driven by how many tokens the (usually) vendor-provided interface (or model itself) wants to use. Then your usage is limited by time opaque time windows.

You call it goofy, in a different context we would call that a dark pattern, shady, prone to fraud

AI subscription pricing was fine when it was $100/month for some opaque 5 hour token budget I don't think I ever used, not even that one day where I coded for 14 hours non-stop using Fable. But like most people with low token usage, I had a human in the loop and and I didn't use workflows with swarms of agents.

Now, of course, the plan is to remove Fable from the subscription. To paraphrase Darth Vader, they have altered the deal. Pray they do not alter it further.

Gpt 5.6 is still like this at least for the $200/month option. It’s also always faster than fabel. Fabel might be able to do some things better but I don’t have time to constantly wait and find out.

They're probably not going to remove Fable. They just extended it for another month.

> tried Kimi K3 on a task I've done with every other model I use regularly and found it chewed a lot longer on the problem and ate up almost the entirety of a 5 hour usage limit on their $19 plan

ArtificialAnalysis puts Kimi K3 just below DeepSeek v4 & GLM 5.2 in token use per task, which is about 2x to 3x more tokens than Grok 4.5: https://x.com/ArtificialAnlys/status/2077832879187620192 / https://archive.vn/zBbFi 2 other open weights MiMo v2.5 & MiniMax M3 are comparatively thrifty.

> Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare

I always put my coding subscriptions (that allow it) through "AI gateways" (Cloudflare & OpenRouter are free) which help track token use.

In my experience, Kimi & Qwen Cloud have opaque & restrictive limits, their "credits" drain faster. I now make it a point of subscribing (directly [0]) with providers that are transparent like MiniMax, DeepSeek, Xiaomi, & Z.ai.

[0] OpenCode Go, Cline, and AtlasCloud have generous limits for open weights, otherwise.

Is Kimi K3 subsidized as hard as the other models out there?

Does it matter? As an end user I really only care about 1) how much I can do in a week, and 2) how long each task takes.

Subsidies would affect 1, but not 2. But if some VC wants to subsidize my Claude or Codex or whatever, awesome.

It doesn't matter if you can switch easily. It might matter if there are barriers to switching.

The more important question than subsidy is what is the tokenomics of running the model. If it's inefficient to run on an nvl72 cluster (or whatever the heck has enough vram to run a 3T parameter model), and k3 isn't very token efficient, then it might not be that compelling of an open weights model.

3T at nxfp4 (which is most of it) is only 1.5TB of vram - so 8x288GB B300 or MI355 will do it if you are careful with context - maybe dp-attn? Certainly not TP. 2 of those together can easily serve it. The new AMD MI400 are at 400GB+ each, so 8x of them will nicely fit with KV to spare.

Subsidization could affect both of those. If you have $200B in the bank you can afford to throw massive compute at every single request; if you are less well funded, you might throttle more aggressively.

Additionally that same VC could be (read: is always) spent on developing the harness, and other infrastructure around the model, not just the model itself.

So it's apples-to-oranges when comparing a relatively new model to established competitors (i.e. OpenAI @ $900B funding vs Moonshot/Kimi's $30B FYI) because every new model they release is judged on "performance" which is not strictly speaking derived solely from the model.

It's possible Moonshot could get similar performance over time as the build out the rest of the infrastructure. We have no way of knowing how much of OpenAI/Anthropic's success is due to the model vs intelligent tooling built on top of it.

Not sure how the economics work for the Chinese models, but DeepSeek did the same task for a dime.

In my opinion, for the vast majority of use cases, DeepSeek is still the most cost-effective model by a mile. $10 feels like it lasts forever.

Yep, with Reasonix, DeepSeek is free real estate. Seems to just go and go for pennies.

And, DeepSeek is what I use for any task that works best with an API. It's cheap enough to where I don't think about cost, made even cheaper by DeepSeek having the most effective and cheap caching in the industry, and it's good enough to where I rarely have to follow up with a more expensive model or manually fix things. It's been alleged they're releasing an update to DeepSeek V4 Pro soon that improves it, which likely makes it a good fit for even more kinds of problems. It remains my favorite of the Chinese models, it's so cheap and cheerful. And, is also less aggressively censored than some of them.

> It's cheap enough to where I don't think about cost, made even cheaper by DeepSeek having the most effective and cheap caching in the industry

I use DeepSeek v4 Flash & MiMo v2.5 Pro. Prefer the latter over DeepSeek v4 Pro because it costs the same while being equally good & less chattier for coding workloads. Although, I've begun experimenting with Hy3 (as an in-between Flash & Pro) & GLM 5.2 (for long-horizon tasks).

Maybe I’m not pushing them hard enough but I use Claude opus at work and deepseek v4 flash at home and they both seem about as capable. While deepseek is borderline free.

From the end of the month it will be served profitably by other providers around the world, like Kimi K2.6

Sure, but at those API rates it doesn't beat the subscription plans for OpenAI.

Absolutely do not pay for the kimi plans thinking they will be cheaper. If you sign up with a Chinese phone number, you can get the same plan for 200 yuan instead of 200 usd, it also only accepts Chinese payment methods iirc. So the plans are really made for Chinese userbase.

how do we get chinese phone numbers?

esim

Nah that won't work. I don't know tbh, I just used someone else's number.

how do I find someone to use their number when I'm not in china and don't know anyone in china?

not gonna happen. China has strict real name verification for SIMs.

Wow! Does it accept a foreign alipay/wechat pay account?

Those exist?

I've had an Alipay account since 2006 and never been to China.

Of course they exist, Alipay is from Alibaba, think about who typically buys from OEMs/suppliers there...

No idea lol, didn't even know those exist..

YMMV, but I'm a US citizen with a KYC'd Alipay account and I was able to subscribe for the Chinese price...

... but I borrowed a friend's +86 phone number, which you'll need to even see that price. or maybe a 回国 VPN will work.

That sounds complicated. I'll just use my month of Kimi and then cancel. I have too many AI subscriptions to use them all, anyway. I subscribed mostly to test it. I mean, if it turned out to be competitive, I would keep it, but if it doesn't turn out to really excel and anything and also take longer than Claude or OpenAI models, I'll stick with them.

It is complicated, but paying for the cheaper usd plans really don't get you much usage.

Kimi K3 only supports "max" reasoning effort right now, but they plan to enable other levels soon [1].

When I looked at traces from benchmarking, I saw a lot of backtracking and uncertainty while reasoning ("wait, but..."). This also happens with GPT 5.6 and Fable with xhigh/max thinking, albeit to a lesser degree.

I think that explains part of the token inefficiency. Hopefully it will improve with lower reasoning effort settings.

[1]: https://platform.kimi.ai/docs/guide/use-thinking-effort

Do models know when they're being benchmarked? I also used K3 briefly and noticed it spent a very long time thinking and obviously a lot of tokens.

However I've seen some benchmarks say it uses fewer than fable which hasn't been my experience.

OpenAI measures token efficiency. Look at the API cost charts in their announcement: https://openai.com/index/gpt-5-6/

It would be really interesting to redo the public benchmarks for kimi k3 but token normalize the costs. Ok so maybe k3 beats fable on terminal bench, but how many tokens did it use?

Aren't they still locking reasoning to "max" pending adjustments to support shorter reasoning levels.

Why would they do that? Sounds terrible

It let's me choose different thinking levels in Kimi Code. Not sure if it actually works, yet, but it says "Thinking set to high." when I change it from max.

Found it!

> At launch, Kimi K3 will use max thinking effort by default, with low- and high-effort modes to be introduced in subsequent updates https://www.kimi.com/blog/kimi-k3

With the obligatory disclaimer that I’m impressed with what open weight models can do, I have the same experience with all of them.

The benchmarks come out and say they’re as good as Opus from N months ago, then I use it for a complex task and it doesn’t work as well as Opus from N months ago did when on similar problems.

There’s a real wow factor when you get an open weights model to do amazing things, but in my experience the gap to the frontier models has always been bigger than the benchmarks would lead me to believe.

There can be a lot of value in having the cheaper open weight models for chewing through lower complexity tasks (non-programming in my primary use case) at a cheaper rate than OpenAI or other frontier API costs. Even with those I can measure bigger gaps to the frontier models than the benchmarks suggest.

If the benchmarks aren’t being directly gamed, there’s at least some selection happening where training data or model structures are being picked in ways to maximize public benchmark performance. All of the labs know there’s immense value in having good benchmarks to show for your model because most LLM consumers are picking based on lab provided benchmark charts, not running their own evals. Running your own evals is hard and expensive.