Hacker News Reader

Comment by gradus_ad | Hacker News Reader

I wonder how long NVIDIA can justify its current market cap once people realize just how cheap it is to run inference on these models given that LLM performance is plateauing, LLM's as a whole are becoming commoditized, and compute demand for training will drop off a cliff sooner than people expect.

View on HN

> LLM performance is plateauing

It’s a wee bit early to call this. Let’s see what the top labs release in the next year or two, yeah?

GPT-4 was released only 15 months ago, which was about 3 years after GPT-3 was released.

These things don’t happen overnight, and many multi-year efforts are currently in the works, especially starting last year.

As someone else points out, training is slightly more involved, but I also find that these smaller models are next to worthless compared to the larger ones.

There are probably some situations where it suffices to use a small model, but for most purposes, I'd prefer to use the state of the art, and I'm eager for that state to progress a little more.

> but for most purposes, I'd prefer to use the state of the art

I'm guessing in the future, we'll see a lot more automatic inference "on our behalf", and you won't care or notice if its using Llama5:3b or whatever comes out then.

I'm betting that in a few years we'll see LLMs baked into a ton of stuff - lots of "simple" things mostly, like email summary/rewording, and similar "light touch" use cases. Maybe light multi-modal work like photo labeling. Probably expanded to be running in every other SaaS applications doing god knows what. For those, the small models would be more than enough, and much cheaper to run in large volumes.

I'm guessing "chat with a bot to ask questions" will be a small amount of the inference that happens on our behalf, but will use the valuable SOTA model use case.

I am a partial believer that the real race for many tech players is actually AGI and ASI later and till the problem is solved the hardware arm race will keep being part of it.

Not only big tech is part of it but billion dollars startups are popping everywhere from China to US and Middle East.

Unless we hit another AI winter. We might get to the point where the hardware just can't give better returns and have to wait another 20 years for the next leap forward. We're still orders of magnitude away from the human brain.

It’s actually about training, not inference. You can’t do training on commodity gpus but yeah once someone figures that out, nvdia could crash

Nvidia doesn’t obviously have a strong inference play right now for a widely-deployed small model. For a model that really needs a 4090, maybe. But for a model that can run on a Coral chip or an M1/M2/M3 or whatever Intel or AMD’s latest little AI engines can do? This market has plenty of players, and Nvidia doesn’t seem to be anywhere near the lead except insofar as it’s a little bit easier to run the software on CUDA.

I know, my point is that when training demand decreases people will be realize that inference does not make up the difference

Yeah the big question I’m struggling with is exactly when training demand will fall if at all

Well every other tech company is writing checks to prove they have the chops to make an LLM. From IBM to Databricks to the big guys like Google. Tons of companies made one just to show off to investors or their CEO or just because they wanted skin in the game. But that probably won't continue forever. We've already seen certain orgs that seem to outshine others, and if they can't catch up, they may just accept to using the open-access models or APIs instead.

At some point everyone will realize that it is becoming a commodity, and it is very expensive to train, then only those with wither a structural advantage to lower price (eg Google) or a true goal of being on the high-end/SOTA of the market (OpenAI, Anthropic) will keep going.

You're talking about training the foundation models. But what about all the fine tuning on non-public business data that will be necessary to make gen AI useful for actual business processes?

I'm finding it difficult to estimate the size of this workload compared to continued training of foundation models. Perhaps it depends on whether there are new architectural breakthroughs that require retraining of foundation models.

And what about non-language tasks such as interpreting video and 3D sensory data? This is potentially huge, but between huge peaks there is often a valley of unknowable depth and breadth.

I have yet to hear a use-case for “fine tuning from business data” that relied on large models to succeed. Once again, I’m skeptical the average business will need this.

Yea yes video probably requires a lot of GPUs to train. And a lot of source material to train against. And a use case. Which again, most companies don’t have.

Model development is clearly here to stay, and clearly valuable. Models from every other company, either foundation or fine tuned - I’m not sure that emperor is wearing many clothes any time soon.

Every research lab is focused on new architectures that would reduce training costs.

Yeah we need essentially hadoop for llm training

> I wonder how long NVIDIA can justify its current market cap once people realize just how cheap it is to run inference on these models given that LLM performance is plateauing

next wave driving demand can be actual new products developed on LLMs. There are very few usecases currently well developed besides chatbots, but potential is very large.

Reply on news.ycombinator.com