What about do you mean by single threaded? Each token is predicted by using parallel computation on the GPU.
Multiple agents need tokens. Should optimize for that instead of one agent blocking the others.
One agent typically blocks the others on a local device because the GPU is already completely utilized either in terms of memory or compute. You can have true parallelism at home, but you need an absurd amount of resources. It's not a simple threading problem.
The typical bottleneck to wider batching on consumer hardware is memory capacity for the KV-cache, not compute (even unified memory/iGPU-based platforms have enough compute to allow for some batching, and SSD offloading changes the scenario entirely). Qwen models tend to have bulky KV-caches for any given token count. But agentic swarms might end up sharing a large cache prefix, so there's scope for potential gains there.
I have no problem running two or three sequences of qwen 27B with a 3090. It's basically the recommended way, LLM inference without batching is super inefficient.
Yes, but I haven’t seen it in Unsloth/llama.cpp. I see it in Sglang and vllm. Unsloth should default to sglang imo
Fortunately llama.cpp works well with 4 concurrent requests now. This is the default, and you can increase or decrease it with -np N