Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU | Hacker News Reader

Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU

piszczek.pl

38 pointspich36 comments

"Combining them into one heroic speedup would make a better headline and a worse benchmark."

"The machine immediately taught me that capacity estimates are just admission tickets."

"Useful in production, poison in a kernel comparison."

Please don't publish writing like this, it's exhausting to read. You can edit that stuff out.

The best part of this is the "Five xhigh artifacts from the finished model" section at the bottom, I suggest either moving that up or at least prominently promoting it at the top of the article.

It has gotten so much worse over the last month. The default writing style of the Claude 5 model series in Claude Code is some sort of jiberish jargon.

I find the ‘explanatory’ output style of Claude to be a bit more tolerable, but yes.

Claude seems to speak and write more in Claude-speak with every release.

Apparently we've blown way past the Turing test and approaching AGI and yet LLM-generated text still sticks out like a sore thumb. Maybe LLMs aren't that good at writing after all.

I think it's fair to say they're better at a paragraph or so than most humans. And have been for quite some time, which is probably why their use in writing has exploded.

Long form though? Still pretty bad. Probably getting worse in practice, as people have them write larger and larger chunks of text without paying any more attention to the result.

Their use in writing has exploded because they can produce a ton of content quickly and people don’t want to put in the effort themselves, not because they are better than humans.

LLMs are great at writing, it's The Assistant who is a terrible writer. Sadly that one persona is all you get these days.

lot of tech folks think their writing is boring and ask AI to make it more interesting or whatever

llm gets them some of that sweet_llm_padding and readers smell the slop and attack the writing :)

people, go write "boring stuff" - its more interesting!

Can you please try and see how many tokens you get with some form of concurrency. Pretty much ALL the benchmarks I've seen on the more accessible cards are just single request.

Because concurrency with a single "accessible" card quickly diminishes. I have dual 3090s, and on Qwen 3.6 35B A3B at 80k a single card with max concurrent set to 4 will get 70-80tps single request, 50-60 TOTAL tps with 2, 45-50 with 3, and around 40tps with all 4 going.

I know this doesn't exactly match your request, but I'm happy with my 3.6's performance. I've seen people claim they can get 120+tps on a 3090, but I'm not impatient when it comes to streaming text faster than I can read it.

You have something misconfigured then. Concurrency has never lowered my overall TPS. Also have dual 3090s. Generally use vllm though.

I've had some rough time getting LM-Studio properly configured for multi-card. It exists, but I feel like it is kind of buggy. I will disable a card and it will still load the model into it. Sometimes it will split the model even though there is loads of room available.

I might need to finally make the switch away from it, but it is so convenient, especially as a chat interface for system prompt experimentation.

vLLM is probably the key difference there… its scheduler is built around batching/concurrency, while this setup is heavily optimized llama.cpp for single-stream latency

Your configuration is broken or wrong. What are you using? Hopefully not llama.cpp?

I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.

What do you use instead of llama.cpp? With vllm for example most models don't seem to be supported out of the box.

I haven’t tried any larger models, but a 12B model on my Ampere A5000 gets around 4-5x the aggregate throughput with concurrency. I have the maximum context configured to 32k, but my actual requests are usually around 2-4k tokens.

No idea how that compares to running a larger model and context though.

I guess it's due to testing on MoE. Different completions activate different experts, thus very little cache reuse and completions "steal" memory bandwidth from each other.

As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass.

With dense Qwen 27B on 3090/llama.cpp I get:

  - no MTP: 1x42, 2x33, 3x24, 4x19 t/s  
  - MTP: 1x50, 2x30, 3x33, 4x30 t/s

That is interesting. I'll have to test that theory out today.

Always put the quantisation in the title!

Its not quite that simple here. The iMatrix-guided hybrid uses different quantization levels per tensor/layer, so there isnt one honest Q4/Q5/NVFP4 label I can put in the title

The whole site looks like and reads like AI slop. The outcomes also don't make any sense and don't feel rigorously tested (no, having claude test for you doesn't count as rigorous).

The person is having a AI induced manic episode, we have all been there.

Please stop making this comment. The war is lost. Instead, you should be commenting that it looks like a human wrote this when you come across the rare brain-produced writing

Why it's flagged?

quantization level?

Its egregious the quant level isnt disclosed along the "Qwen" string. Everyone knows theres huge difference in speed/quality along the quant axis, I now attribute the ommision of such to deliberate choice to not curb the hype of the tittle.

The article mentions Q4, Q5, Q8, and NVFP4. It's total AI slop though, tough read.

In my testing I got 150 tokens/sec with a single 5090 RTX.

Which model/quant/command line did you use? I can barely get 100 token/secs and for sure clearly not a full context. With vllm, I am limited to 130k tokens with vllm + nvfp4.

If you've got RTX 5090, maybe try ninfer (https://github.com/Neroued/ninfer). Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer (NVFP4; 256k ctx).

Looks like slope benchmarks and results, and as usually, people are mixing MTP numbers with non MTP numbers. Or just 100 token input benchmarks. Or just failed ones as actual measures.

https://github.com/Neroued/ninfer/blob/master/docs/performan...

Category MTP3 stochastic sampler DFlash stochastic sampler DFlash greedy Code 1/15 natural stops; 0/15 prompt-complete 2/15 natural stops; 0/15 prompt-complete 0/15 natural stops Story 9/15 natural stops; the nine Chinese outputs pass requested division and minimum length 8/15 natural stops; the eight Chinese outputs pass requested division and minimum length 10/15 natural stops; five Chinese dialogue outputs are under length Translation 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks Structured 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract

And on my "own" "quick" benchmark, it's slower than vllm.

I don't have a 5090, so I can't really comment, but here's the relevant reddit thread from today where they report the numbers (including ninfer ones), and where you can make your case: https://www.reddit.com/r/LocalLLaMA/comments/1vqjeub/how_man...

yeah, I was confused during the whole thing, i get 70 t/s on a 3090, which evens out around 50 t/s at 128k+ , have been running 3.6 and now 3.8 (both iq4_nl at 256k q4 kv) on the 3090 for months. I am confused as to what we 'discovered' here, it's a common config. and at less than 1/2 the price of the gpu (and double the bandwidth, though no fp4 cores to be fair).

A 3090 has 936 GB/s of bandwidth vs 432 GB/s on the 70W RTX PRO 4000 SFF, so 70 t/s there is not surprising. The interesting constraint here was fitting a workload-tuned 5.01 BPW quant + 256K + MTP into 24 GB while working with less than half the memory bandwidth

they talk about the quants they tried in the article and settle on a Qwen3.8-27B-Hermes-iMatrix-NVFP4-Balanced.gguf which they calibrated on their own session traces and they pulled in 5 different llama.cpp pull requests to their local llama-server.

5.01 BPW custom hybrid: bulk NVFP4, selected Q5_K/Q6_K tensors from an iMatrix, Q6_K embeddings and Q8_0 lm_head. The iMatrix was built from 5,472 messages across 296 real Hermes sessions

Glad to read theyre not 296 fake Hermes sessions /s

Reply on news.ycombinator.com