That said, if someone would kindly quantise this down for the 64GB paupers, that would be appreciated. (I know there’s likely degradation, but some people reported good results with a 2 bit version of Qwen 3.5 122B, and this is starting from a higher point. Would be interesting to try, at least.)
Edit: someone in the process of doing so: https://huggingface.co/vcruz305/Laguna-S-2.1-GGUF
https://github.com/vllm-project/llm-compressor
my setup to help you on your way: https://github.com/verdverm/quantr
Though it seems these will not be needed as Poolside has published quants & dflash with their models.
https://huggingface.co/poolside/Laguna-XS-2.1-GGUF/tree/main
Maybe the chat template is broken, maybe not. But I can't get it to think/reason, or at least sometimes it seems to decide not to even when asked. And it is wildly inconsistent when it does it, sometimes thinking out loud.
However, without thinking enabled it is shockingly good at researching using web tool calls. It's also very fast.
I am assuming something is not yet right with the chat template, even using a recent build of llama.cpp, and I guess I will revisit this model again at some point in the future.
The Q4_K_M is 75GB. I'm exactly at 64GB, and I wouldn't quantize it further. Instead, do partial weight residency and stream the rest from SSD.
In the huggingface link they mention building for CPU and for CUDA, does anyone know if that means it wouldn’t be possible to build targeting Vulkan?
Unfortunately I don’t know about the best way of running on an Nvidia gpu, you could try llama.cpp and offloading as many layers as possible into the gpu and using RAM for the rest, not sure if that would slow it down too much though.
Someone has benchmarked a wide range of different quants.