Hacker News Reader

Comment by simonw | Hacker News Reader

The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models

I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.

View on HN

Depending on which model you're running, you might need to use the custom forks.

Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....

I spent quite sometime trying to install their tools and nothing really worked. I used these repos you shared but the dependencies all fail on mac

If you can share details on where it is failing, we'd love to help fix.

You can also join Discord to communicate with us directly http://discord.gg/prismml

I did not know you guys would be watching. For sure. Let me do that tomorrow when I turn it on again :) I am happy to see the message. Thanks!

I can report that it's working in oMLX. I've been experimenting with the ternary one; it is quite an impressive model! I've been grilling it on some deep learning/computer vision stuff and it's aced everything so far. Responses are thorough, accurate, sophisticated. General knowledge outside of CS doesn't seem as robust, which I expected. Honestly, I don't think the examples in the blog post do it justice.

Nice, I tried it too with oMLX — agreed, it seems very capable for coding! Was slightly underwhelmed by performance though. I got about ~24 t/s on the ternary version on my M2 Max 64. That’s quite a bit slower than Qwen A3B 35B (4 bit unsloth). How was perf for you?

I had roughly the same performance on M3 Max 64: ~30 tokens/sec. Which is not terrible, but with the latest Lightning MTP optimization I am getting ~100 tokens/sec with Qwen 3.6 35B-A3B.

WHAT. that‘s amazing. thanks so much for sharing, i‘ve been looking for ways to speed a3b up for days. It‘s 11pm here but i‘ll try this right now

I’m using the new oQe quants in case that matters.

i see! would you mind sharing your model string?

i'm getting similar toks on my 36gb m3 max. new omlx release (0.5.2) has dedicated decode kernels for the bonsai models — see if this improves perf for you.

I got their previous model working in their custom fork of llama.cpp (https://github.com/PrismML-Eng/llama.cpp). I haven't tried this one yet, but will find some time to benchmark it sometime this week.

Though this says mainline llama.cpp has their patches for Metal and CPU backends, so maybe it's simply "use current llama.cpp" if you have a Mac or fast enough CPU/memor to use the CPU backend.

Not sure if there's any way to run Prism's fork of llama.cpp inside LM Studio.

The fork runs fine for me. The model gets very notably stuck in a reasoning loop on one of my simple tests, though it might be that it has the same issues with setting reasoning effort high.

On my M1 Max I still think the MoE Qwen 3.6 and Gemma 4 models are the best options. And I am far from convinced that the 35B is actually worse; it gets stuck in reasoning loops much less often than 27B in my experience.

Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.

Have Prism-ML upstreamed their forked code?

Yeah it's a llama.cpp fork and it's linked everywhere on their repo

Of course — I compiled it and ran it yesterday. Not aware if they have a PR in place with the main project, though as I recall I ran the last Bonsai model without needing it so I guess there might be plans.

I downloaded two of the official ones in LM Studio, both 3.6gb, and neither loaded.

Reply on news.ycombinator.com