1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project.
2. The depth anything comparison isn't apples to apples. Of course a q8_0 quant is faster than f32. It's 4x less data to chew on.
3. This point is silly, it again fights against the argument that writing your own c++ engines are worth it. It's a bug, just fix it:
> The reason it is faster has nothing to do with writing better matmul kernels than PyTorch. Two positional embeddings, the DPT head’s UV embedding and the backbone’s bicubic position embedding, were being recomputed on every forward pass with single-threaded scalar sin, cos and bicubic loops, even though they depend only on the input geometry and are identical every call.
They argue against their own point again just after:
> For a biometric pipeline, matching the reference exactly matters more than being faster than it.
Okay, then don't rewrite it! It's not faster anyway!
4. If the argument is that the venv is large, then rewriting it in C++ doesn't seem like the answer, it seems like a lot of work and maintenance to avoid having to cull unreachable files in your venv. In a past life I maintained a simple denylist for files in node_modules. The low hanging fruit is plentiful and generally very safe.