Hacker News Reader

Comment by nblgbg | Hacker News Reader

Is there any advantage to using the model from Unsloth compared with https://huggingface.co/Qwen/Qwen3.8-27B-FP8 ?

Depends on what software/hardware you'll run it. GGUFs from Unsloth can run on pretty much every single potato; full weights need beefy gpus

We also made NVFP4 ones if that helps! https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4

This is the version we'll be testing on our rtx 6000 today! Thank you

Why not just run FP8 on vLLM with that much vRAM? It's plenty fast.

For high concurrency, using the blackwell's native native W4A4 MLP compute path, nvfp4 is something like a 1.2-1.5x performance increase over FP8. We're doing data enrichment (so, tasks completed successfully + tokens/second) so the performance bump shows up in the tasks/month number.

I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.

I see. Did you see any intelligence degradation between FP8 and NVFP4 for 3.6 27B? You're using vLLM, right?

I didn't have time to run this as well, but we're getting a 99% agent completion rate across all tasks and 98% task decision that matches the human selected option(s) on Qwen 3.8 27B @ NVFP4. There may be a difference between FP8 and NVFP4 but it's inconsequential for our data enrichment purposes.

Unsloth one is gguf for llama.cpp (and some other on-device engines).

So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.

Run the unsloth if you are using llama.cpp (GGUF)

Run the one you linked if you are running vllm (safetensors)

Unsloth usually also fixes the models when they bork something, which always happens. For Gemma for example the tool calling wasn't working for the longest time.

That wasn't our problem right? Gemma officially updated tool calling which we adopted

if you have the VRAM, use offical release. quantized model lose focus after long context and can do damages or thinking loop

Reply on news.ycombinator.com