Hacker News Reader

Comment by noodlesUK | Hacker News Reader

I feel like the centralisation of server resources for something like copilot doesn’t really make sense. Many (most) professional developers are working on beefy laptops. If ever there were a case to run these models client side, a software developer’s laptop is probably the ideal place. What are the specs that are needed to run inference on these models?

View on HN

There's a huge range of CPU and memory requirements to run these models. The lower end of the range can be run locally but performance is noticeably worse. I'm guessing this will change as more people are looking at efficiency now. Specialized hardware like Apple's Neural Engine on the MacBook Pro may also help.

> I feel like the centralisation of server resources for something like copilot doesn’t really make sense.

For whom? For those selling them it absolutely does, because that's how they get to charge a huge ongoing markup on hardware and electricity costs. That's why everything is a service now.

I don't think most dev laptops have 8GB+ of GPU memory, which (based on ollama requirements) seems like it's on the low-mid end of the requirements. I've tried experimenting with some local models and:

a) They're much slower on my 6GB laptop GPU

b) The seem to not be as good, functionally

b) I can't make use of larger models

I haven't done more than just some experimentation but I can see how this would make sense to put on a server.

MacBooks especially pros are in a very privileged position due to their memory architecture. With some dedicated hardware it may be reasonable to run, perhaps even train, useful LLMs on device.

I guess we’ll see in a year or two. This must be on everyone’s radar now, Apple won’t be the odd man out.

AMD is probably best positioned to make a laptop/desktop SoC with the same kind of unified memory. Though I wouldn't be surprised if Intel's little experiment in budget dedicated GPUs is also a byproduct of similar R&D.

Lisa Su was interviewed at Code recently, and the AI discussion focused on competing against NVIDIA in big datacenters, but I hope they're also thinking about client-side stuff like that.

At 600$+/y subscription fees, it makes sense to own a beefy NUC with 32Gb and a speedy SSD to run the model locally.

I would hope developers value their time at least 100USD/hour. So unless the local option is less than 5 hours per year of maintenance, the subscription makes sense.

One's time is free. Many developpers make less than 3k$ /m net, so it still makes sense.

The secondary problem to the high resource requirements, is that I think once you ship the model people can just basically pirate it right?

I’d be surprised to see the models go the ground breaking commercial AIs shipping but then I’m surprised at how much we’ve gotten open source anyway.

A lot higher than the beefy laptop... Even a 4090 can't run it well.

Reply on news.ycombinator.com