Hacker News Reader

Comment by endlessvoid94 | Hacker News Reader

Hm, I've got an M2 air w/ 24GB. Running the 27B model was crawling. Maybe I had something misconfigured.

No, that sounds right. 24GB isn’t enough to feasibly run 27B parameters. The rule of thumb is approximately 1GB of ram per billion parameters.

Someone in another comment on this post mentioned using one of the micro models (Qwen 0.6B I think?) and having decent results. Maybe you can try that and then progressively move upwards?

EDIT: “Queen” -> “Qwen”

You also need to leave space for other apps. If you run a 27B model on a 32GB machine you may find that you can't productively run other apps.

I have 64GB and I can only just fit a bunch of Firefox and VS Code windows at the same time as running a 27B model.

That rule of thumb is only related to 8 bit quants at low context. The default for ollama is 4 bit, which puts it roughly about 14GB.

The vast majority of people run between 4-6 bit depending on system capability. The extra accuracy above 6 tends to not be worth it relative to the performance hit.

I think only 2/3 of ram is allocated to be available to the gpu, so like 14gb which is probably not enough to run even Q4 quant.

This is configurable by the way.

sudo sysctl iogpu.wired_limit_mb=12345

deepseek-r1:8b screams on my 12gb gpu. gemma3:12b-it-qat runs just fine, a little faster than I can read. Once you exceed GPU ram it offloads a lot of the model to the CPU and splitting between gpu and cpu is dramatically (80? 95%?) slower

Reply on news.ycombinator.com