Hacker News Reader

Comment by Hamuko | Hacker News Reader

I've tried to use a local LLM on an M4 Pro machine and it's quite painful. Not surprised that people into LLMs would pay for tokens instead of trying to force their poor MacBooks to do it.

View on HN

Local LLM inference is all about memory bandwidth, and an M4 pro only has about the same as a Strix Halo or DGX Spark. That's why the older ultras are popular with the local LLM crowd.

Qwen 3.5 35B-A3B and 27B have changed the game for me. I expect we'll see something comparable to Sonnet 4.6 running locally sometime this year.

Could be, but it likely won't be able to support the massive context window required for performance on par with sonnet 4.6

This would be an absolute game changer for me. I am dictating this text now on a local model and I think this is the way to go. I want to have everything locally. I'm not opposed to AI in general or LLMs in general, but I think that sending everything over the pond is a no-go. And even if it were European, I still wouldn't want to send everything to some data center and so on. So I think this is a good, it would be a good development and I think I would even buy an Apple device for the first time since the iPod just for that.

I’m super happy with it for embedding, image recog, and semantic video segmentation tasks.

What are the other specs and how's your setup look? You need a minimum of 24GB of RAM for it to run 16GB or less models.

This is typically true.

And while it is stupid slow, you can run models of hard drive or swap space. You wouldn’t do it normally, but it can be done to check an answer in one model versus another.

48 GB MacBook Pro. All of the models I've tried have been slow and also offered terrible results.

Try a software called TG Pro lets you override fan settings, Apple likes to let your Mac burn in an inferno before the fans kick in. It gives me more consistent throughput. I have less RAM than you and I can run some smaller models just fine, with reasonable performance. GPT20b was one.

Tokens per second is abysmal no matter how much ram you have

Some models run worse than others but I have gotten reasonable performance on my M4 Pro with 24 GB of RAM

Local LLMs are useful for stuff like tool calling

What models are you using? I’ve found that SOTA Claudes outperform even gpt-5.2 so hard on this that it’s cheaper to just use Sonnet because num output tokens to solve problem is so much lower that TCO is lower. I’m in SF where home power is 54¢/kWh.

Sonnet is so fast too. GPT-5.2 needs reasoning tuned up to get tool calling reliable and Qwen3 Coder Next wasn’t close. I haven’t tried Qwen3.5-A3B. Hearing rave reviews though.

If you’re using successfully some model knowing that alone is very helpful to me.

Reply on news.ycombinator.com