Hacker News Reader

Comment by sandworm101 | Hacker News Reader

But that per-token cost is a total joke. All these companies are fighting to build market share in some future dominated by one or two AI ecosystems. It is musical chairs until someone creates the one ring to rule them all. So they are charging token amounts just to claim revenue as they burn through investor dollars.

In short: per-token charges currently cover maybe 1% of the total costs in this field. To pay ongoing costs, and pay back investors, everyone will need to pay 100x or 1000x the current rates, likely for decades.

View on HN

> In short: per-token charges currently cover maybe 1% of the total costs in this field

There are plenty of seemingly informed people saying the exact opposite, so that's a lot of confidence you're talking with. I have a hard time believing it when we know what open weights models cost to run. And sure, there's training costs, but again many say inference costs are already above training costs.

If that's true, it's very unsustainable.

Gemma-4 26B-A4B + M5 MacBook Pro + OpenCode isn't Claude Code _yet_, but it's good enough that if I were forced to use it I would be fine.

Yes, it's amazing how quickly so many tech companies have hitched their tooling to these big AI vendors seemingly without any thought towards whether they'll still exist a year or three or five from now. Insane behavior. To the (debatable!) extent that AI coding tools are useful at all wouldn't it be a hell of a lot smarter to self-host? At least that way you have some control over QoS, and a stable, predictable result... Or maybe nobody cares about that kind of thing anymore? What happened to basic business math in this industry?

The basic business math is (to start) software companies realizing that spending $10k, $20k, $50k (more ?) per year, per developer for current models at current token rates might not be particularly insane, given the value return.

Models are likely going to keep getting better, and as costs go down, demand is likely to rise faster.

> as costs go down

Huh? Why would that happen? Indications are that costs will likely go up, especially if currently vendors are selling tokens at a loss.

The main operational expense of a million LLM tokens is pennies of electricity.

Even if you generously depreciate the GPU and other hardware, it’s hard to believe inference at scale in April 2026 isn’t highly profitable.

> The main operational expense of a million LLM tokens is pennies of electricity.

I think you meant dollars of electricity.

I don’t think so.

https://www.theregister.com/2024/03/18/nvidia_turns_up_the_a...

A Blackwell 8X node consumes about 15kw, let’s up that to 50kw to generously account for cooling and everything else.

A US kWh is something like $0.20, so running that node for an hour costs ~$10.

Nvidia got 30,000 parallel TPS out of DeepSeek-R1 on that node:

https://developer.nvidia.com/blog/nvidia-blackwell-delivers-...

So that $10 buys you over 100M tokens or … pennies per million.

I’m sure these numbers are off, but not by an aggregate two orders of magnitude.

It’s getting better on both the hardware and the software fronts the barbarians are banging at the gates.

I'm not sure this information is grounded, but I remember to have read somewhere the inference is indeed profitable. My personal experience is similar. Running 2x3090s draw 500-600W and you can locally run amazing models with a similar setup.

Running the model isnt the cost. Watts per token is the math they show investors. You also have to be constantly training new models, which currently needs more compute than servicing the customer base. You have to biuld datacenters, and possibly powerplants to feed them. You have to carry debts. And you will need to buy new GPUs/ram every few years to remain competative. The total business is vastly different than simple gpu math.

You are in violent agreement.

> inference is indeed profitable

From the perspective of a deal like this, “total costs in the field” matter less than incremental cost per token served.

The unit economics for today’s frontier models should be great, and this suggests Anthropic believes they’ll get better.

In a decade the cost of compute will be a tiny fraction of what it costs now. Specialized hardware will exist that will be cheap and efficient.

The difference in the cost of compute between 2026 and 2036 won’t be nearly as large as the difference in the cost of compute between 2016 and 2026. Even at 2016 the slowdown in improvements was noticeable.

We might see a one time bump in inference when we move off GPUs onto more limited and efficient dedicated hardware, but the sustained fast pace of improvements are far behind us.

I'm predicting now that there is a clear use-case for this tech that work will (and has) accelerate specialized hardware, software, models, etc that will run much more efficiently in 10 years. So that the real token costs will be a fraction of what they are now.

You can run models on FPGAs and get massive cost, speed, and throughput gains (like 10x). The reason people don’t do it is because of other improvements (algorithmic) means that nobody really thinks locking into a model makes sense…yet. Would I want to use gpt 4o for anything today at 1/10th the price? That would be $0.40 per input, $1.50 per output. Gemma-4 31b is much more capable and cheaper. So a FPGA version of the model is just not worth it today.

But if progress begins to slow down, then the economics work. Maybe Gemma 4 is a good example. It feels really generally useful. Getting it at 1/10th the cost feels like it could be competitive in 2 years.

The fpga would be for prototyping. The real progress comes from asics ... exactly as we saw with bitcoin mining. This GPU-based approach will eventually give way to bespoke circuits once everyone picks a favorite model.

Compute power improvement between 2016 and 2026 wasn't that impressive either. Moore's law is essentially dying.

Yeah I went shopping for a new computer a couple of years ago (to replace a 7 year old computer) and... the specs for what was for sale were the same as what I bought 7 years prior, and the price wasn't much lower.

I would much rather buy a 2026 computer than a 2019 computer. Two generations of Nvidia GPUs, Apple M series chips, the X3D AMD chips, and pcie5 ssds are all major upgrades.

It’s just that the pace of new stuff is slowing down, and many people are operating under the assumption that this wave will ride on forever.

Reply on news.ycombinator.com