Beats Opus 4.7 Max (w/ Claude Code) on DeepSWE (42.2 vs 40). Looks like Qwen's 27B models continue to pack some punch.
Unsloth's GGUF quants are up: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
Unsloth's GGUF quants are up: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.
Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.
I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.
There are tools like promptfoo designed for this.
That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria.
It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."
We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.
For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.
This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.
As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.
0% is 4 percentage points (pp) less than 4%.
I can say I find the law stupid, so no one should say person A beat person B in court. But I did not prove the law is stupid; I merely thought it subjectively and demanded others to follow the second part because I believe the first part is true.
Saying that "if the law is useless, court cases are useless" is objectively true and cannot be argued with. But you still need to prove why the law is useless, not why you think it is and even then if people disagree and use the law as a reference, then it's not objectively useless and court cases are not useless as well.
deepseek-v4-flash needs web search to return true facts.
For example you would tell a model hey, become an expert in this language for me, search it online, it would still need to learn it and download the data to it's context and then increasing the memory usage, there's no way around it.
Ideal local model would not know stuff like who Britney Spears is, best to leave precious weights for something useful.
Of course the line is very blurry but I'd be perfectly happy with local model that doesn't know anything about history, geopolitics, art or even biology etc. just coding, operating systems etc.
So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.
Run the one you linked if you are running vllm (safetensors)
I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.
It's a bit bare at the moment, I assume they are going to add further detail later (eg comparison to other quants), similar to their other releases.
Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.
It also has a lot of resolution and not a lot of noise. Better would be multi-turn benchmarks with tools but getting good precision and accuracy for that is hard and computationally expensive.
You can't really know that either.
Sometimes they're just slow and expensive, so we we KLD as a proxy measure and it's very high correlation (95%+)
Gemma 4 31B: "Um, if I really said all of that, I guess I'd say this next"
Gemma 4 26B: "Dude, I would've said completely different stuff" (large divergence)
Gemma 4 12B: "Umm, there's zero chance I would've said some of this" (INFINITE divergence)
Gemma 4 E4B and E2B: "Derp derp, I'm happy to say almost anything" (lowest divergence)
For models which are chat trained, they simply would not recite Wikipedia, so the divergence is almost meaningless. I thought about capturing a realistic coding session and trying to use that as the corpus, but you need to preserve the turn-based tokens and such, so I moved on to other things.
e.g. If you try to chat to it about something philosophical for example, or maybe a debate / creative writing, then you'll very quickly see how it is still a much smaller model at the end of the day.
Still, it's such a relatively accessible model to run, and I find a big part of leveraging smaller models is to give it well-scoped tasks; not too high level or ambitious ones. Very impressive for its size and the ability to run locally :)
I am currently using Qwen 3.6 on RTX 3090 and I have to admit that without MTP it would be too slow to be acceptable for me (30-35 tok/sec without MTP, 60-70 with MTP). Without MTP I would just use OpenRouter and rather pay for speed despite having a capable local setup.