Hacker News Reader

Comment by rTX5CMRXIfFG | Hacker News Reader

Affordability of hardware that can run local LLMs is a real factor, too. Not sure when RAM prices are going down, but with everything that’s happening and can happen in the world right now, it doesn’t look like it’ll drop in the near or medium-term

View on HN

No one is going to run models that are comparable to frontier locally without spending enormous sums for use at scale or in large orgs. Even with cheap RAM, you will still need a very large budget for frontier-level capability.

Open models that are competitive with frontier will be used on shared hosts.

Models have been capped out on training and (active) parameters a while ago, its tooling / harness that is making the big jumps in performance happen. And then you have things like DeepSeek with a pretty small KV cache.

And with the extreme chip shortages for the next two years, there's little appetite for even bigger models anyway.

Barring a breakthrough in scaling, the only direction the models can really go is smaller. Which will inevitably mean better performing local models for same chip budget.

> No one is going to run models that are comparable to frontier locally without spending enormous sums for use at scale

You can always run these models cheaper locally if you're willing to compromise on total throughput and speed of inference. For most end-user or small-scale business needs, you don't really need a lot of either.

It would be awful if running models locally became the primary way of using LLMs. On dedicated servers sharing GPUs across requests, energy usage and environmental impact is way lower overall than if everyone and their mother suddenly needs beefy GPUs. It’s the equivalent of everyone commuting alone in their own car instead of a train picking up hundreds at once.

You can batch requests when running locally too, if you're using a model with low-enough requirements for KV-cache; essentially targeting the same resource efficiencies that the big providers rely on. This is useful since it gives you more compute throughput "for free" during decode, even when running on very limited hardware.

That’s still orders of magnitude less efficient, and also not how most people use AI, or probably will use AI.

It's even more awful if the compute capital is owned by only a handful of players.

Maybe people would target their use more appropriately, then.

Just like people would drive their car as little as possible out of concern for the environment..?

So instead, we are building data centers for capacity we aren't sure exists...

Data centers that are orders of magnitude more resource intensive than anything than came before. Hell there is one planned for Utah that I saw would consume 2x the power of the states current usage, which would there by triple a single states power usage overnight.

Tell me how that is somehow more efficient?

It's not. But the choice isn't "use cloud AI from a data center" vs. "do not use AI." Way too much money has been sunk into the technology, and it is far too seductive to ignore, so AI is here to stay.

So the question is rather what the most efficient way to serve AI is - locally or from the cloud, so: data centres.

Open weight models does not means you can run them on your laptop (except for the small ones). It means that someone independent (a cloud provider, another company ...) can build big computers that are capable ton run those models and provide you a metered usage.

At the end of the day, as a consumer, you still pay per token (or per something) to your provider, except you can chose from multiple providers with your own criteria. If you want to use DeepSeek v4 hosted in Europe, it's possible.

In other words commodification, and no moat.

Which would also be an ideal outcome for people interested in avoiding a concentration of power and wealth due to access to generative AI.

Reply on news.ycombinator.com