Hacker News Reader

Comment by FinnLobsien | Hacker News Reader

You can set spending limits, but I don't feel like that helps much because everyone's accustomed to AI. Nobody would accept "We're out of usage so we have to wait until Monday" and do all of their work manually.

I believe that we're in a scenario where usage is unlikely to go down and neither are frontier AI costs.

I believe we'll see a shift to more organizations building their own harnesses with model routing logic to get central control over who can use what AI and for what.

A marketer doesn't need to default to Opus 5.5 to upload a blog article with MCP, which could be done by a model 10% of the price.

View on HN

It's not just being accustomed to AI. It's that the adoption of "agentic" workflows can make teams dependent on AI.

This summer my team basically shut down when we hit a usage cap because we had recently pushed a decent portion of our devops workflow into agent skills. It had been done in such a way that it was difficult for humans to navigate. Relevant scripts turned out to be buggy and poorly documented, and nobody realized because these harnesses that are tuned to be absurdly tenacious about searching for workarounds had been quietly burning heaps of tokens on muddling through instead of raising any alerts about the horribly broken state of the system.

I do agree that home (or at least independently) grown harnesses might be the next logical step. Harness vendors who charge by the token have an inescapable conflict of interest here. Moderate, well-governed LLM usage isn't good for their revenue.

True, I guess I was mostly thinking of people using AI in their workflow, not just AI code review and things of that nature.

> I do agree that home (or at least independently) grown harnesses might be the next logical step. Harness vendors who charge by the token have an inescapable conflict of interest here. Moderate, well-governed LLM usage isn't good for their revenue.

It's true that moderate usage harms current harness providers, but I think that's because we currently conceptualize them as AI services. I think in the near future, companies will have harnesses that centrally configure MCPs, CLIs, model routing, etc.

and they'll be much closer to an auth/permission service than to an AI service.

I have some bias as a software dev/manager but at least in our org I noticed you can get a lot better value out of your tokens with some proper configuration of context and skills. Providing the latest models with good context can result in extreme savings.

After we started excessively documenting and extracting skills everyone has stopped complaining about running out of tokens, because agents stopped having to reconstruct the full context each time from scratch. Harnesses like claude code also push the model to aggressively keep this documentation in sync so there's little concern about drift.

The worst thing to do from a token usage pov is to give a model a vague open ended prompt because they are so scared to be wrong that they'll waste a ton of tokens "thinking" through the issue and verifying everything. Whereas they almost trust skills blindly and skip all this unnecessary work.

No one would tell truck drivers "We hit our gas budget for this week, deliver the rest of the packages on foot."

Well no, but even with the gas prices jumping the way they are, that is a much more predictable cost than AI spend. Drive 250km, in 2015 Volvo truck is X liters of fuel at X cost. Solve a predefined problem using AI... you have no idea what that will cost you, if you did you'd most likely already have the answer.

No, but truck drivers (who work for a company making this decision) do sometimes get told there is no work for them because nobody wants to pay the price. (this also happens when there is nothing to ship though - drivers can't tell the difference)

No, but I think this is wrong on three levels:

1. Truck drivers are not deciding to take a Ferrari instead of their truck. The company knows roughly how much gas a given delivery is going to consume and how much that is.

With AI, that definitely happens if you default to Claude Opus/GPT Sol.

2. A truck driver's value scales with gas usage. The further they drive, the more valuable.

Not true for tokens.

3. Gas doesn't create extreme outliers. One developer can easily consume 100x the tokens of another, which makes AI economics much more extreme.

> I believe we'll see a shift to more organizations building their own harnesses with model routing logic to get central control over who can use what AI and for what.

I can also see middlemen agencies coming into the picture to determine token use, best model, etc. Effectively a middleman to the LLMs that will sell token use on the popular models bundled together.

My company setup tiers (as of this month - I can't tell you how it works since this is only day 5). I automatically get so many tokens, if I use them up I'm auto approved to upgrade to the next tier. They told everybody wait until you run out to request a tier increase - the real goal is a few people had a runaway job burning tokens that they didn't stop, the tier is a force stop point in case that happens to you.

Yeah I feel like token usage is probably power law distributed as well.

Also this feels like a good way to detect runaway jobs. I’ve seen something similar implemented for data plans where you had unlimited data, but every 50 GB had to request another free 50 GB to prevent people from turning their phone plan into their home WiFi

Well at a big bank I work, if you finish in 1 week you do manual coding the rest of the month until next month. Management doesn’t give a shit. And they have daily talks about using more AI.

Banks have historically invested as little as possible in technology, unless it directly generates cash (fintech). I'm not too surprised.

.... that sounds hilariously ineffective

so you basically are rationing your frontier tokens?

This has happened before with cloud computing. There was lots of people talking about how every employee would spin up their own servers etc and huge productivity gains. In the end all of that leverage mostly got absorbed by the engineering teams and gatekept and the rest of the business gets to use it through a GUI.

> I believe we'll see a shift to more organizations building their own harnesses with model routing logic to get central control over who can use what AI and for what.

This is what happened with cloud computing. People with expertise wrote software around the software creating machine because just giving naked compute to people en-masse didn’t actually achieve anything valuable. Likewise all of the talk of enterprise agents etc, they’re all custom software that wraps around a reasoning model.

> This is what happened with cloud computing. People with expertise wrote software around the software creating machine because just giving naked compute to people en-masse didn’t actually achieve anything valuable. Likewise all of the talk of enterprise agents etc, they’re all custom software that wraps around a reasoning model.

Yeah exactly. Plus, most people don't want to configure MCPs and write skills about how to correctly use CLIs. The type of person who uses Hacker News may do that to customize whatever harness they use.

A sales manager at a sheet metal firm in Wisconsin doesn't. They want AI tools that make their lives easier, but they don't know or care whether that happens with an MCP.

I guess the cloud is a great analogy because that same sales manager may have loved Salesforce because it was always available and updated compared to a manual spreadsheet.

Does the sales manager even care if it's AI either post hype? For most people on the "AI assistant" use case they care about having something that they can type a question into and smart answers come out. If you could serve it with a cache and call it AI they wouldn't care if it's not giving dumb answers.

Reply on news.ycombinator.com