The harness that connects to a chatbot, API or voice interaction is the place to route requests to different systems. If you remember the early days of ChatGPT it explicitly said it was routing image generation to Dall-E after embellishing your request itself first.

Determining which tool to use should be a lightweight operation but I’m not expert enough to understand exactly how much lighter than a full LLM call just to recognize it needs a different tool or model.