My experience mirrors this: services like OpenRouter that promise “failover” are pretty much useless except for sandbox testing because models in production are not really interchangeable. Any production harness doing serious agentic work in production is dependent on more model-specific quirks than you would expect. And even if another model works without errors, performance and efficiency is a whole different story. Even the system prompt can and should be tuned to a model’s preferred speaking style, for example <xml tags> for Claude-like models because they were trained on it, while other models do better with other delimiters. Think of the whole harness, prompt, and model as one system, not really with modular parts that can be swapped out if you care about optimal performance.