My experience with AI orchestration and LLM routing

I've encountered many AI orchestration platforms that facilitate multiple AI model integration and management. For example, nexos.ai provides access to over 200 AI models from leading providers, including OpenAI, Anthropic, Google, and Meta. It offers features like smart model routing, intelligent caching, and comprehensive monitoring to optimize performance and costs.

While these platforms simplify testing different models without altering code, I haven't found a strong need for a dedicated router. I tend to consolidate tasks into a single prompt, selecting a model capable of handling the most complex aspects, which naturally manages the simpler parts. Therefore, I have fewer "easy" prompts, as straightforward tasks like text fix-up and routing are minimal.

My approach to "routing" involves directing inputs to specific prompts, each with its own context. Since the combined context exceeds capacity, routing becomes necessary. However, this concept differs from traditional model routing.

Another consideration is the portability of LLMs concerning tools, functions, and structured outputs. While models like Opus and Gemini Pro 1.5 have made progress, GPT has been a leader in this area. I often use these features with smaller prompts within larger algorithms to avoid the complexities of text parsing and handling exceptions from unstructured outputs.

Portability in LLMs, especially when it comes to tools, functions, and structured outputs, is a crucial aspect to evaluate as well. We see advancements with models like Gemini Pro 1.5 and Opus but still GPT stands out as a frontrunner. While models such as Opus and Gemini Pro 1.5 have shown advancements, GPT stands out as a frontrunner. I learned to incorporate these features with more concise prompts within larger algorithms. This helps to minimize the difficulties of dealing with errors from unstructured response and text parsing.

Ultimately, as I'm not particularly sensitive to pricing in my work, I gravitate towards the latest GPT models. If I were to switch to a platform like nexos.ai, it would be for enhanced functionality rather than cost savings. I prefer making thoughtful decisions to update the default model in my code, minimizing the need for frequent adjustments.

Given the advancements in AI orchestration platforms like nexos.ai and others, how do you see the role of LLM routers evolving in balancing cost, performance, and model specialization?