Hacker News Reader

Comment by baxtr | Hacker News Reader

While this might be true I’m worried about the hardware side of things.

What if you have a good enough model but the cloud model providers are better in procuring hardware for interference?

Local inference is definitely going to make more and more sense. Modern CPUs have all this amazing hardware well-optimized for inference purposes. I use a lot of web tools and see AI baked in and it feels weird. I want the smartness localized for speed and data security. I think and hope the industry points towards smart ai agents operating as locally as possible.

The cloud providers are probably better at procuring hardware for inference, but on prem users are better at repurposing hardware that they'd need anyway for their existing uses. In a world where AI compute is likely inherently scarce, it makes sense to rely on both.

I personally believe that eventually manufacturers will want to sell more of their hardware and look for ways to sell hardware to consumers. isnt that situation quite similar to the days of early computers? I am for sure biased in hoping that will be the case

Perhaps for some very specific capabilities such as TTS, translation, voice recognition and so on. But for general intelligence models, better hardware just directly allows better models and that doesn't seem to be changing any time soon.

I'm pretty sure that's not linear, so I personally expect the benefits of larger models to diminish. The question is at what point that's the case. I guess a lot of variables play into it, but it is possible that the benefits of running larger models will be too expensive for the little benefit they provide

You’ll be able to run the open models on any cloud at the cost of the hardware rental. While the closed models will try to mark up beyond the base cost.

Reply on news.ycombinator.com