Hacker News Reader

Comment by LeoPanthera | Hacker News Reader

...but you can't run it locally. Not unless you're sitting on some monster metal. It's tiresome when people compare enormous cloud models to tiny little things. They're completely different.

View on HN

> ...but you can't run it locally. Not unless you're sitting on some monster metal.

I'm getting a very usable ~18 tok/s running it on 2x NVIDIA A4000 (32GB VRAM).

Both GPUs cost less than USD $1,400 on eBay.

qwen2.5-coder:32b is 19GB on ollama [1]

[1] https://ollama.com/library/qwen2.5-coder:32b

I believe that parent comment was referring to the point that Sonnet 3.5 cannot be run locally which it obviously cannot but not because of locally available compute but because it's not OSS.

or probably both? Seriously doubt 3.5 Sonnet is in the same order of magnitude as 32B

Right, it's almost certainly a magnitude larger than 32B.

I'm curious if the M4 Max will be good enough.

That will run Qwen 2.5 Coder 32B just fine. I'm using a M2 Max with 64GB of RAM.

How many tokens per second?

With MLX I got:

    Prompt: 49 tokens, 95.691 tokens-per-sec
    Generation: 723 tokens, 10.016 tokens-per-sec

Yeah, over 65GB VRAM... that'd be expensive but not impossible. I think three RTX 4090's could do it, with their 24GB each.

Only 21.3GB is required.

Is this wrong? It says 65.8GB. If it's wrong, what source should I be using instead?

https://llm.extractum.io/model/Qwen%2FQwen2.5-Coder-32B-Inst...

the ollama one is probably quantised.

Ollama's default quantization is q4_0 which is quite bad, but you can go to ollama's model page to see all the quantizations they have available, you can do e.g. "ollama run qwen2.5-coder:32b-instruct-q8_0" which will need ~35G + space for context

... with 4-bit quant and 4000 token context.

You really don't need models to fit into GPU's anymore for inference, e.g. llama.cpp is pretty good at partial GPU offload and I've gotten pretty fast results with only being able to fit ~30% of a model in VRAM and the rest in DDR5.

Reply on news.ycombinator.com