Hacker News Reader

You can't afford the best model. What are your specs and what models + quants have you tried?

Qwen 3.6 35B A3B and Qwen 3.6 27B can both do reliable tool calls on Pi at Q4_K_M using llama.cpp

I'm on a 48 GB M5 Macbook Pro. I use 4-bit quants with a context window of 16-32k. I tried Qwen 3.6 27B, but I can only get around 10 tokens per second, but it's painfully slow, and it often fails during `write_file` tool calls, even with Qwen Code.

Try an 8 bit quant of Qwen 35B, but temper your expectations. Those Qwen 3.6 models are impressive for the size, but you need an order of magnitude more parameters to actually be useful for more than trivial work in my opinion.

Reply on news.ycombinator.com