How do you prepare the fallback if it already took tokens to query the simpler model?
I don't know if I understood your question.
don't litellm and similar do this already?
Absolutely, a pretty good one.
Actually, LiteLLM is the most popular, it was first product of this type, but there is plenty of more efficient and reliable alternate AI Gateways right now. I've written one - GoModel. https://gomodel.enterpilot.io/