Ways to think about token pricing | Hacker News Reader

Ways to think about token pricing

ben-evans.com

46 pointsmercutio222 comments

1. Slot machine

You put in minutes of time combined with token costs. You pull the lever and HOPE something good comes back.

And it gets tantalizingly close, but for 100% gotta pull the arm again!

that is way too accurate and ofc hn would downvote something funny and true.

it just seems off-topic. we see the slot machine analogy a lot but i dont see how it has anything to do with this long, thoughtful essay about pricing power

Pricing in the article is the difference between a $.25 slot machine with a range up to the $10 slot machine.

Its always a probabilistic gamble that you get what you describe. And it's a pull of the lever each time. And you pay no matter what, for good or bad results.

Its gambling. Either your time or time+money.

Tokens will surely become commodities, but models don't need to.

The point is that electricity becomes a commodity, rather than, say, nuclear reactors or gas turbines also need to be commoditized.

Contrary to most people in the software-minded circle, I am not very excited about running local LLMs. Most of what I have seen involves selling wrappers that call APIs on top of an LLM AI, similar to how traditional SaaS has generated revenue from new technologies. So, this would certainly make those people who are working on that sensitive about the pricing. LLM pricing is treated as a continuous cost in the COGS. They don't really use AI to create much value (consumer surplus) other than replacing the existing boring products, like just building the good old CRUD apps, but faster.

What I find REAL interesting about the potential of LLM AI instead is to create new technology out of it, or revolutionize something old to be order-of-magnitude more efficient. In this regard, the expenses on those tokens are more akin to an upfront CapEx.

Cheaper tokens would surely be nice. But if what we are talking about is, like, solving self-driving, curing cancer, or making air-conditioners 100x efficient, the narrow focus on running a cheaper model in my home so I can write my SaaS apps for my get-rich-quick business looks really unpromising and a waste of time for human civilization in a near-singularity horizon.

Agreed, although most of what humans do is pretty pointless in that context. Even if AI turns out to be a multiplier that makes everyone quicker and smarter, people will mostly just want to use it to make more money than the next guy. The history of capitalism suggests that the money-go-round will create useful technology along the way.

I know there's a lot of reasons to think that everyone will just use AI inference in the cloud, but I think if everyone had access to a dgx gb400 class machine with 512gb of hbm4 vram and 1tb of lpddr8x a lot of people are going to be running finetunes of models locally. Like the dgx gb300 is $94k now, but i bet this class of machine will come down to $20kish in the next 2-3 years.

sure, if there werent capitalists digging their moat. in 1 year local inferwnce doubled cost from ram alone because cloud vendors cornered markets on their circular cash flow expectations.

Yeah but Samsung/Micron/SK hynix are building new fabs, and Chinese Ram will be able to fully supply the lower end ddr5 ram market within 2 years. The crunch is temporary and on the other side of it there will be incredible hardware at the $10-20k price point.

There’s a shitload of money to be made in RAM fabrication. Costs will come down a ton.

sure, and there more in becoming a diamond cartel. you have yoo much optimism for real market forces.

RAM industry is larger.

You are waaaaaay too optimistic about RAM prices. This is not a one-quarter crisis and you need to settle in for the long haul: the era of "hardware always comes down in price" is over, definitely for the next 3 years and possibly the next 5-8. The new DRAM fabs aren't even coming online until 2028 and there's not much reason to think they'll make much of a dent in the supply-demand mismatch even when they do.

DRAM prices have been highly cyclical for a long time. They'll come down again.

From what I'm reading on the shortage, they are very aware of this fact (prices being highly cyclical) and they're trying to avoid overproducing RAM to avoid the oversupply glut. That's probably going to hurt the chance of prices coming down anytime soon.

Prices will come down when the stupid money isn't all going to datacenter buildouts.

Maybe micron/samsung/sk won't expand enough, but trust me the Chinese will. It might mean we don't get access to HBM for a while, but you can do a lot with infinite cheap DDR5

The marginal cost of making RAM is zero and it's clear to everyone that there's near infinite demand, so the fab capacity is coming. Maybe the high end won't expand fast enough, but you can do a lot with Chinese ddr5.

if all the competition pushes down margins on tokens to 10-20%, i dont see how the inherent scale advantages of cloud inference wont be way more to account for the 20% cheaper tokens youd be getting running locally. i dont see how local will ever be more economical

What is the economic cost of having your data monitored by external sources?

I agree with the author's overall conclusion about pricing LLMs, although I question some of the reasoning. He does not talk enough about one important input: the data used to train the models. To a first approximation, their utility is a function of the quantity of data available. Currently, the bulk of the training data is public (the internet) and that is a major reason why the performance of different models is not diverging more. One huge future area for AI is modeling the physical world and in that arena people are going to be creating their own empirical datasets. Waymo has achieved FSD and Tesla has not. It is not coincidental that Waymo has built a sophisticated platform to simulate the physical world.

price per token on its own tells you nothing.. what matters is tokens per successful outcome a cheap model that needs three retries costs more than an expensive one that gets it right first go.

anyone actually tracking cost per completed task rather than cost per call ? it is quite a difficult task to define completion as it is a qualitative aspect and can vary from every individual.

Reply on news.ycombinator.com