Hacker News Reader

Comment by smeeth | Hacker News Reader

1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model.

2. quantization != native low precision training. a model trained in native ternary should generally outperform a full-precision model quantized after the fact.

even if a ternary model only retains 90-95% of the performance of its fp16 equivalent, who cares? if a 200b ternary model retains most of the capability of the 200b fp16 model while using a fraction of the memory and bandwidth, it can be substantially less efficient per parameter and still dominate a smaller fp16 model under the same hardware budget.

View on HN

> yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance

I know that's what the paper says the benchmarks say, but these models feel significantly worse than the base model when you start using them for real tasks.

Even the Q4 quant which they put in between their Bonsai models and the FP16 in the benchmarks has a tendency to go into doom loops and get lost compared to even Q5 or Q6.

I don't know how much of this is due to benchmaxxing (putting the benchmarks into the post-training loop) or cherry picking benchmarks to look good. If you spend a lot of time using local models you learn to take vendor provided benchmarks with a huge heap of doubt. Everything looks amazing in the benchmarks these days.

Models are not like a fps counter where losing a few percentage has no actual impact. These percentages may be the difference between a model that writes code and one that goes into loops, and think rm -rf . is a good idea.

There is a reason why most models try to stay in the FP4 or higher range, because the reduced accuracy can have major consequences.

You are better off with a 8b FP4+ model then a 27b Q2 model.

You're conflating two concepts: native bit width and post-training quantization.

Consider two models: one is 16B and trained natively in 2 bits; one is 8B and trained natively in FP4. These models have the same total number of weight bits, but one has twice as many parameters. There is no real reason the FP4 one should be better just because it's FP4. It might be, but that is an empirical question, not a general rule.

Post-training quantization is another thing entirely. Taking a model trained at higher precision and forcing it down to 2 bits is going to hurt performance, often very badly. But this was never my point.

has anyone been able to get bonsai models to run at a proper speed in a x86 windows cpu-only setting, using llama.cpp?

Reply on news.ycombinator.com