Hacker News Reader

Comment by sanderjd | Hacker News Reader

I think the article's analysis is basically right in a vacuum. That is, I think it's clear that inference is a viable business model. But what isn't clear is whether it will be such a profitable business model for any given company that it will justify the investment that company has taken. I kind of think the winners might be a follow-on generation of companies that focus on this commodity inference business model instead of the invent-machine-god-first "business model" and thus are wiser about their level of investment and capital costs.

View on HN

> I think it's clear that inference is a viable business model.

You may be right. I'm not so sure. Inference looks like a viable business model for those operators that have SOTA infrastructure in place, but the investment required to have it is enormous, and appears to be never-ending, because if an operator stops investing aggressively, its infrastructure quickly becomes non-competitive, and customers will quickly leave for alternatives. SOTA infrastructure is a moving target.

Maybe... I'm not enough of an expert on the financials to say, but it seems to me that inference should be able to recoup the cost of SOTA infrastructure, unless you then also use a large portion of that infrastructure to train new models. And I also think the race to remain SOTA itself is also largely a function of training, because my understanding is that training benefits more from the leading edge of hardware.

But yeah, I definitely don't have high confidence in any of this!

Yes, that makes sense. I don't have "the" answers either :-)

What about inference on a distilled version of someone else's model? What about inference on an open weights model? I don't think all LLM business models are going to revolve around developing state of the art frontier models.

Right. This is what I was implying in my comment, but it's good to be explicit about it. I think there will be successful companies that just sell inference against the best models they can get without needing to invest anything (or very little) in training anything new. This could even be a spinoff of one of the big frontier labs. I would personally rather invest in an IPO for a company that only owns the gpt-6-astra implementation and infrastructure than in openai itself. There is certainly less upside, but IMO also way less downside risk. (I'm not saying there is any chance this kind of a spin-off would happen, it's just a thought experiment.) And I think this same calculation applies to open weights inference providers.

Edit to add: Or or might just be AWS / GCP / Azure that benefits from this business model. They're already pretty good at selling commodity infrastructure.

Renting out compute is a viable business model. That's part of what the big and the small cloud providers do. It's just (on its own) not a "get rich quick" business model anymore.

Why would it be any different for inference? If we believe OP, it'll just become part of regular compute infra, and thus part of the renting-out-compute business model.

I think it's an open question if the current generation of inference investment will pan out, but in the long term, there'll be a balance between investment cost and margin, just as in every other industry.

> I think it's clear that inference is a viable business model.

Only if you also have the model thats better than anyone else's.

As soon as models are free, or there are no newer models (assuming thats going to happen, and thats not a given) then the only thing you can compete on is price.

This means that the only thing you have to differentiate is either price, speed or ease of use. (or regulatory capture...)

We are at pets.com level of spend currently. Unless model development becomes cheaper, then we are going to run out of novel debt but not really debt mechanisms.

I mostly agree with this!

But I also think there are multiple ways to differentiate. There is at the very least: "intelligence", price, latency, throughput, reliability. It's not clear to me yet what this looks like, but maybe there is also a services and integration level of differentiation. And then there is the universal stuff: sales, marketing, branding. And then on the other side of the ledger there is operational efficiency, management capability, cost of capital, that kind of stuff.

I mean, there is no kind of "model quality" difference between AWS and GCP or between Delta and Southwest or between Wal-Mart and Costco, etc. but all of these businesses remain viable in very competitive markets.

I totally agree that the level of investment / capex is not sustainable though! But I think what's going to happen is that it is not going to be sustained, while AI continues past that point as a viable business (but maybe with different specific companies leading that industry).

A phrase comes to mind: "Your margin is my opportunity."

Reply on news.ycombinator.com