adchurch

Effectively yes (based on cost though, not raw token count)

We trained a model to select which LLM to call at any given turn, based on lots of agent traces

Yes the open source models are very good, that’s a big part of what makes this router save so much money in practice! There definitely are some things they still don’t handle well though where you do want a frontier model

Yes we can route to Gemini models too and we handle all the translation complexity there!

We welcome the competition :)

Yep exactly

Yes because it's a model explicitly trained to make model selections! Opus probably doesn't have a great idea of when to send a task to DeepSeek vs. to Sonnet, for example.

We haven't experimented with routing to local LLMs much. Technically they benefit from the cache too although it's more a question of latency than cost. But tbh I haven't seen great results in the wild from working with local LLMs for coding - curious if you've had any success with them?

I think the key detail here is that we use embeddings of the prompt + previous context in order to decide where to route the request, and if one model is getting stuck we can course correct and move to a different model.

So: we can reasonably cluster similar problems together and learn how models handle them, and the entire system doesn't fail if the initial decision is off.

We consider the cost of missing the cache when making each routing decision after the initial one. Discussed in a bit more depth here: https://news.ycombinator.com/item?id=48689448

Good questions. From what I can tell, vLLM semantic router is more optimized for one-off prompt/response workflows rather than agentic coding (I don't think it's cache aware).

As another commenter (https://news.ycombinator.com/item?id=48689994) pointed out, for one-off requests, I think it makes more sense to lock to one model whose behavior you understand very well. For dynamic requests like the ones going to a coding agent I think dynamic routing makes more sense but it does need to be cache aware.

Cool, interested to see your approach when you do launch! I think it's a really interesting problem

Great question! Our main product quantifies engineering productivity & quality so I think we're uniquely qualified to answer this - our velocity has only gone up and our quality (bugs introduced, code turnover) has not budged per our own analysis.

Oh interesting, didn't know Cursor did that! Totally makes sense though, routing subagents is def the easiest win, no need to have any cache awareness.

If you have a Claude sub with subsidized usage we use that. If not you pay API prices.

Really appreciate the thoughtful feedback!

1. Agree it's important, fwiw the proxy model doesn't blow this up though - only incurs a 1 time cost when switching models and we're aware of that when making routing decisions

2. The agents are model aware yes but they are not incentivized to optimize too heavily here (in particular they don't use OS models even when they would be better). I think that's where this router comes in and brings genuine improvement.

3. Two parts here: 1 is continuing to grow our golden dataset over time, 2 is using reward signals from production traffic (on a per-customer basis or, if allowed, across all users)

4. Yes we have these internally, great callout that we should publish! Will do + will link from the repo soon. (Fwiw I think these benchmarks are useful but don't fully capture vibes - you should try it out yourself for that!)

Appreciate the kind words! Lmk if you have any feedback on it from using!

I would argue they do not have a good incentive to build this and make it better. Why would Anthropic route Claude Code traffic to DeepSeek (at 20% of the cost)?

Very important consideration, addressed it in another thread (https://news.ycombinator.com/item?id=48689448). tl;dr we built this to be cache aware for exactly this reason

When we started building this we did it as an experiment and we thought the same thing might be true (cache misses would make the whole thing pointless). This turned out not to be true! I think there are 3 reasons intuitively:

1. Small models can carry out a good number of requests e2e 2. Small model for part of a request + cache miss < big model for entire request in many cases 3. Subagents

For our own usage we've saved 40% so far (that is of course including costs of uncached requests when switching models)

Fair enough, not meant to be marketing just a statement of fact. Would have turned me off too 18 months ago but times change...

It's a real concern! We take this stuff super seriously (https://trust.mycroft.io/weave) and tbh most of our customers opt for the hosted version because it's much simpler on their end + they're already trusting us with a bunch of sensitive data.

But of course since the source is available you can also run it locally or self host

Yep cache awareness is super important, mentioned this in another thread here: (https://news.ycombinator.com/item?id=48689448)

But intuitively I think it makes sense that a model can learn what model to route things to if it has all the relevant info, and experimentally it works pretty well in our experience

I guess delivering business value is always #1, I just meant it's the biggest problem they're trying to solve. Here's a recent example that was public: https://fortune.com/2026/05/26/uber-coo-ai-spending-tokens-c...

The choice on the first turn is super important for this reason! But if a user prompt sends the convo in a very different direction then often it does make sense to reroute at that point.

Nice, let me know any feedback you have from trying it out!

Very interesting - curious how you've used it yourself so far? I can imagine one use case would be having e.g. GPT 5.5 review Opus 4.8's work?

Yep 100%, mentioned this in another thread (https://news.ycombinator.com/item?id=48689448) but tl;dr we build the router to be cache aware

Indeed it is :) I skipped over talking about all the RL machinery, network design, reward function design, state representations, etc. because really the intuition is that we tell the model when it accomplishes its goal, and then it learns over time how to get better at making the right decisions in order to accomplish its goal.

Happy to talk about this in some more depth if there's anything specific you're curious about!

But we're not routing via `claude -p`, if you have sub usage available + it's the right choice to route to a Claude model, then the router is approximately a transparent passthrough. So it gets billed like normal `claude` usage rather than `claude -p`.