Hacker News Reader

Does the router approach make sense with multi-tasking LLMs? One could execute multiple different tasks with a single prompt (chat response, NER, translation etc.) and with the latest models even do images or video alongside text. Doesn't a router get in the way, unnecessarily increasing latency?

View on HN

So the neural scoring introduces ~20ms latency, but this only impacts time-to-first-token (not inter-token-latency). When using our public endpoints there is an additional ~150ms latency, but you can deploy the router on-prem in your own cloud, so then it would only be the inference latency. Generally the improvements in ITL outweigh the small addition to TTFT.

Reply on news.ycombinator.com