Hacker News Reader

Comment by TeMPOraL | Hacker News Reader

Who are the customers though? Honest question, I'd like to understand it.

For me, anything other than current best available SOTA for any task is unacceptable. The only routing rule I need is "the most powerful model I still have flat-priced quota available for". I mean, why settle for less?

View on HN

There are two types of users: those who are able to use subsidized rates, and those who need to use API rates due to audit requirements, enterprise billing, etc.

Model routing for subsidized users takes the form of a "use Opus 5 subagents for implementation" type of system prompt. You lean into a single provider, build tooling around that, and your savings are far beyond anything multi-provider routing can get you.

Model routing for enterprises is far more complex - approaches like https://fireworks.ai/blog/kimik3-fable become necessary for cost control.

on top of that there is also additional factor: speed - sometimes if task is easy you do care to finish it faster.

There is also matter about convenience - when I ask some small easy question often I don't bother to switch the model or forget in prompt to ask faster/cheaper subagent.

It's very common to use a lesser model for a lesser task, resulting in same quality output. End result: save money while being faster. In many cases, it's a pure win-win.

But in other many cases you have to redo the work directly or indirectly, and you are more expensive (for now) than even the most expensive models, so sounds like a total lose.

I’m assuming you don’t pay per API call. Every mid-large sized business in the world does.

> For me, anything other than current best available SOTA for any task is unacceptable.

Then you must route. An article with lots of upvotes yesterday or two days ago showed that K3+Fable 5 was more SOTA than either of those.

Sometimes you can get an equivalent result in a fraction of the time using a less capable model.

The problem is of course knowing ahead of time that the faster model can give you the same result.

I'm not a customer of those routing systems, but I quite often use different Claude models for different tasks. While most tasks were Opus 4.8, I often used 4.8 to make a plan, prompts, and package kit to setup Fable for a bigger project, then run it on Fable. Or, for broad single-task searches Sonnet with or without "Research []" turned on seemed to work best both faster, lower overhead, and less verbose answers (when I didn't want it).

OFC, YMMV

And you do not have flat priced quota for Fable 5, right? Because nobody has, as far as I know. So you'll probably not route any task to the "current best available SOTA".

Also: quota. Implies you do not have unlimited access even for flat prices. Which in turn implies that as soon as you hit the quota on the most expensive flat price plan, even you will suddenly discover the magic of economically sensible behavior.

Fable 5 is included in the Claude Max subscription. I've gotten close to the limits this week and last but haven't hit it yet.

But Fable is only available for 50% of your Max quota as far as I understand

Not everyone is you. Other people probably have a range of tasks that can accomplished with different models.

Certainly if I'm confident that I'm going to get what I need from a faster model, that's what I want to use, rather than wasting time grinding away for the sake of saying of the same answer came from a SOTA model.

Given that every chatbot does offer a range of models, it seems clear people do choose among options.

The mental effort in estimating what model would be better is so not worth it.

I just want to switch to Claude Code, tell it to turn a .csv into a BigQuery table then cmd+tab to something else while it runs. Thinking "oh this is probably an easy task, I can /model to Sonnet to save $0.0004" is silly.

The game changes when it's not just "Opus vs Sonnet", but "Opus vs GLM". The amount saved is way more than even $0.04. And it's not only money but speed. Some providers can serve GLM crazy fast - I'll even go outside of my subscription to pay extra money for the quick results.

Not everything is interactive, and what the routers do is precisely take away that mental effort on repeated tasks.

People actually use the models for more than writing code.

I barely used Fable because of the rate limits. It just makes more sense to use Opus.

If there were no limits it would be different.

It is not to save a fraction of a penny, it is to be able to still use the model within the limit on the week for $20.

because "less" can be so much faster?

not just that -- "less" ($$$) can also result in indistinguishable quality for some tasks/inputs. I'd argue this is the primary reason, secondary being speed.

I generally agree. Perhaps there's only a 5% chance that it would write better code or find a bug that it wouldn't have with a lesser model, but the economics of bugs is strong enough that preventing a single bug is worth hundreds of dollars.

That's nice and all, as long as flat-priced quota continues to exist. Seems unlikely to go on forever!

Reply on news.ycombinator.com