Hacker News Reader

Comment by jermaustin1 | Hacker News Reader

Because concurrency with a single "accessible" card quickly diminishes. I have dual 3090s, and on Qwen 3.6 35B A3B at 80k a single card with max concurrent set to 4 will get 70-80tps single request, 50-60 TOTAL tps with 2, 45-50 with 3, and around 40tps with all 4 going.

I know this doesn't exactly match your request, but I'm happy with my 3.6's performance. I've seen people claim they can get 120+tps on a 3090, but I'm not impatient when it comes to streaming text faster than I can read it.

View on HN

I guess it's due to testing on MoE. Different completions activate different experts, thus very little cache reuse and completions "steal" memory bandwidth from each other.

As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass.

With dense Qwen 27B on 3090/llama.cpp I get:

  - no MTP: 1x42, 2x33, 3x24, 4x19 t/s  
  - MTP: 1x50, 2x30, 3x33, 4x30 t/s

That is interesting. I'll have to test that theory out today.

You have something misconfigured then. Concurrency has never lowered my overall TPS. Also have dual 3090s. Generally use vllm though.

I've had some rough time getting LM-Studio properly configured for multi-card. It exists, but I feel like it is kind of buggy. I will disable a card and it will still load the model into it. Sometimes it will split the model even though there is loads of room available.

I might need to finally make the switch away from it, but it is so convenient, especially as a chat interface for system prompt experimentation.

vLLM is probably the key difference there… its scheduler is built around batching/concurrency, while this setup is heavily optimized llama.cpp for single-stream latency

I haven’t tried any larger models, but a 12B model on my Ampere A5000 gets around 4-5x the aggregate throughput with concurrency. I have the maximum context configured to 32k, but my actual requests are usually around 2-4k tokens.

No idea how that compares to running a larger model and context though.

Your configuration is broken or wrong. What are you using? Hopefully not llama.cpp?

I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.

What do you use instead of llama.cpp? With vllm for example most models don't seem to be supported out of the box.

Reply on news.ycombinator.com