In the comments it reads like bonsei falls apart on longer running tasks.
Not really. The largest IQ4_XS quant here is still worth it because Bonsai doesn't offer larger quants. They could beat it if they made a quaternary variant though, I don't know why they're stopping at ternary.
I wonder the same thing for Bonsai 2. ByteShape offers 5 models from IQ2_XXS-2.56bpw (8.8GB), IQ3_XXS-2.88bpw (9.9GB), IQ3_XS-3.01bpw (10.4GB), IQ3_S-3.23bpw (11.0GB) to IQ4_XS-3.84bpw (13.1GB). Their benchmarks show gradual improvement with size and users can pick one to fit theirs need. Bonsai-2-27B now is about 8.6GB. It might be good to have a quaternary version around 10-11GB to fit a computer with 16-24GB RAM.
Agree here, as it is ByteShape wins practicality wise if the goal is doing actual work with these quants!
prismml's title is very misleading. in their own paper the model is at 75% of coding scores.