However they do seem very close to SOTA : I’d say roughly equal to previous gen (Opus 4.8, GPT 5.5). It’s yet another silly benchmark, but compare them here: https://senko.net/vibecode-bench/
I also had K3, Qwen3.8 and Fable (using Kimi Code, Qwen Code and Claude Code harnesses respectively, and the official APIs) create a simple but far from trivial web app (zero shot, from a detailed spec). In user testing all three results looked/behaved more or less the same. I had Sol (via codex) do code reviews on all three and it concluded all three were solid, with some room for improvement. Fable was slightly ahead of the pack.
In my own work I still prefer Opus 4.8 (until Antrophic come to their senses and allow 100% Fable usage on Max plans) and Sol, but if I had to find an alternative, I could live with both K3 and Qwen3.8 just fine.
Its hard to compare that one to my regular sub of anthropic (107€), but everyone always says how much cheaper it is so i thought it was worth a try.
1.5 days later ive went through more then 50% of my weekly usage and decided to renew the regular anthropic sub too.
The API Pricing is definitely cheaper, but anthropics subscription budget seems to equalize that advantage right now. also kimi code feels like claude code from ... september 2025
also - considering how they announced they'd temporarily close subscriptions to make sure they can service customers... i was slightly surprised that there was no issue subscribing, less then 1 day after that announcement. Makes you think if that was just a marketing stunt
Let's judge the speed of the model when its weights are released and every inference provider on the planet offers it, so demand can spread out a bit.
It's the same topic with token budget comparisons and subscription pricing - don't people understand that this doesn't really matter for open weights models? The pricing is going to be determined by the inference providers, and until they had a chance to evaluate the model on their infra and set token prices accordingly, one doesn't really have anything tangible to compare with other open models nor with closed ones.
On the other hand I expect K3 future refinements to be massive and more efficient.
Also this arm of the discussion was about speed, not price.
Yoh have posts in this thread suggesting that Fable is 5x more expensive than Kimi K3.
* Kimi K3: 9532k input (9172k cached), 114k output - cost $5.5
* Qwen 3.8 Max: 18020k input (17836k cached), 114k output - cost $6.3
* Fable: ~14m input (all cached??), 196k output - cost $30
Correction on my earlier post, Kimi was through Pi, not Kimi Code. For Qwen I used Qwen Code and for Fable I used Claude Code.
Not sure wtf is going on with the Fable stats (a lot tokens, virtuall all of them were cached - I guess heavy system prompt?) but both claude code stats and ccusage tool output match.
* Kimi K3: (...) cost $6.3
* Fable: (...) cost $30
It's pretty clear that Kimi K3 beats Fable by a long margin.
To objectively test all models the harness would need to be the same and ideally independent. Failing that, all three models should be tested in all three harnesses and the output verified on a model x harness level and on an overall aggregated model x all harnesses level.
Yes they do according to databricks -> https://www.databricks.com/blog/benchmarking-coding-agents-d...
That's why I find comparing models on benchmarks only gives the tendency. We should be comparing model x harness to have accurate metrics.
Testing each lab’s model in its own harness just tells us how well the lab has performed. For the model’s performance we need a control and the only way to get the control is to either test all models in all harnesses or all models in the same harness
This wasn't just being slow - it didn't make forward progress.
For simpler stuff even 2.7 does just fine, though.
The 5.5/5.6 series has performed quite well however. My guess is that my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.
What do you mean by that ? If the model is higher than 50% on swebench pro then it tends to drift from what you like it to do, like DeepSWE benchmarks ?
Kimi was half the cost and produced a near identical output.
On a fairly simple coding task Kimi was 2x the cost and 7x as slow (and gpt-5.6-sol was even cheaper).
This totally depends on the model. Deepseek V4 is very fast and efficient.
But yes, it’s cheap.
People talk about frontier as if it's the only innovation worth pursuing. Deepseek V4 is far from fontier, but it's architecture is super innovative and efficient and what it achieves at that size (especially V4 Flash) is incredible.
Those are all frontier-competitive models.
The product is a fairly standard Ruby on Rails webapp with postgres as the DB. Application complexity is probably a bit higher than average for a webapp. So it's nothing that pushes the boundaries of software engineering, but it is a real product. Token budget has not been an issue for me. I pay for the Max plan ($200/month) and it is well worth it.
I've read the same opinions about Opus and yet it was gpt 5.5 pro via api tackling the hardest problems.
I have used now k3 for 3 days and it has consistently tackled difficult problems sol max could not (orientation optimization algorithms of random 2d shapes on a rectangle for glass cutting).
I have also other beefs with Anthropic models which have been getting smarter and more capable since 4.6, but increasingly worse at acting as assistants, they just want to "do" stuff their own way and ignore instructions often (even simple ones like not to commit, let alone complex ones).
For low-level x86 assembly coding, Fable is nowhere near to be as good as Kimi K3.
We run an evaluation that only compares models in open-ended multi-agent environments where agents affect each other, primarily testing writing code. It's designed to be less vulnerable because there's no solution set, and it's been pretty effective and tends to rank Chinese models lower than their advertised model cards (relative to US models). Kimi K3 is a bit of an exception there -- it truly is a near-frontier model. But it's so slow. Ironically, Muse Spark 1.1 is one of the strongest models we've tested after Fable and Sol while also leading the cost efficiency curve. Big turnaround from Llama 4.
Data at https://gertlabs.com/rankings
I can't actually get Fable to do anything. I only work on back-end code, and the moment Fable notices the jwt scope checks on the endpoints it's game over, it refuses to do anything because security is involved.
So for me, Fable is completely useless, the bar is very low, any llm that will actually attempt the task beats it every time.
For implementing a feature where browser credentials need to be handled securely in the app, Fable refuses to work.
Outside topic, but check out the Inkling model -- it is SUPER FAST and does a good job at being a terminal buddy but I would probably offload the real programming or hard tasks to fable or someone else
Anthropic/OpenAI were touting PhD-level intelligence three years ago. And they’re still shipping models that aren’t smart enough to realize things such as the need to drive the car to the car wash (because they hadn’t yet hill-climbed that particular brain-teaser).
No, they weren't. GPT-5 was where OpenAI started talking about PhD-level, and that was less than a year ago.
(Sept 2024) OpenAI claimed o1 was phd-level in their launch post.
You're kinda both wrong. :)
When this first went viral, I immediately tried it on Opus (whatever version was latest at that time), and it got it first try. Tried a few more times in fresh sessions and it got it every time.
Sonnet did screw it up, though.
"Despite being a highly competitive model overall, K3 nonetheless exhibits a noticeable gap in user experience compared with Claude Fable 5 and GPT 5.6 Sol."
(And, yes, I know we should blame C. C turns every bug into a cybersecurity nightmare.)
I'll see what I can ship while not relying on these closed frontier models at all, as unreasonable as that might be. And if I don't get any customers that's on me.
For both GLM 5.2 and Kimi K3 I feel the rough average of the benchmarks gives you a rough idea where they stand. GLM 5.2 was sitting somewhere behind Opus 4.8 but it didn't feel very far away. I've used Kimi K3 via OpenRouter and while I've had limited experience so far, it sure as shit feels like it's right up there with Sol and Fable to me. I am happily able to believe Fable has the edge still, but on a request by request basis it would be pretty easy to get an impression one way or another.
The existing Kimi models were already pretty good so I really just don't find this new model to be that hard to believe. Maybe I'm naive.
It's better to not even follow the news or benchmarks and just use whatever is available. I make my own judgement.
It's exactly the opposite. Go turn on min_p once it's available post July 27th and most of the problems you describe will go away.
This seems both arrogantly dismissive ("you are holding it wrong") and incorrect.
Either the OP is using Kimi K3 on Moonshot where is is presumable set correctly (K3 isn't available elsewhere yet), or they are using Kimi K2.x and there has been plenty of time to experiment with this.
You can either not believe me and be wrong, or you can (after July 27th) turn on min_p or a better sampler (i.e. top-n-sigma if you got it running via llamacpp) and have it work even better. Up to you.
That would be terrible because Kimi 2.x is in a different world than Kimi 3
Having opinions on current Kimi model based on 2.x would be like dismissing gpt-5.6-sol because of experiences with gpt4o
Assuming your temp is below 2, min_p of 0.1 is fine (and disable top_p and top_k). You can try 0.05 for more diversity.
Remember that subsequent methods are better, min_p is a mid-tier sampler that just happens to be the best implemented in most inference providers right now.
Also I'm the author of the "conspiracy against high temperature sampling" thing that selfhoster posted in the comments, so you can ask any questions about that piece you want.
Just as a call to actual consideration, would that seem a smart bet to do? Because I find it really hard to justify dismissal at this point, sure I don't think these things will improve forever and ever, but come on. The goofy Will Smith spaghetti is a little over 3 years old. Three years. Look at where we are at.