KVarN: Native vLLM backend for KV-cache quantization by Huawei

It's not better quality: 59.3% vs 59.4% fp16 on AIME 25

any divergence (even if the benchmark is better) from full precision is error

Faster than Fp16, not better quality i guess

It's the output of a research paper; the authors are not trying to build up vLLM, and they probably have no incentive to do so. You can submit a PR, though! It's easier now while the divergence is low, so don't wait. Since there are six authors, I bet you could get help with the inevitable review chores if you just take the step of creating the PR.

edit: It might not be clear that it is based on vLLM 0.22, which is the current version: https://github.com/huawei-csl/KVarN/commit/d6290e99098d7426d.... All you have to do is create a diff off it; it's fairly straightforward.

Last I heard, vLLM was backed by a company that has raised $150m in seed funding. I'm sure they've got the resources to port it.

Why this is not a PR for llama.cpp

it should be easy to do btw

This is awesome! Let's give them some stars: - https://github.com/huawei-csl/KVarN (original repo, vLLM implementation) - https://github.com/Anbeeld/beellama.cpp (llama.cpp implementation + awesome evals)