The true value of this will be proven over time with wide use and as PRs are merged.
Do you have a feel if you'll try to drive this to stay feature parity with llama.cpp, or are you willing to diverge with new features like NVME/SSD MoE weight streaming etc.
What I actually want is MoE on machines that can't fit the model in VRAM, and specifically expert-level residency instead of layer offload: track which experts get hit during decode, keep those resident, evict the rest. Doing that well needs the router, the KV cache and the memory manager to be designed together, which is about the only good reason to write a runtime from scratch.
Yes, let’s see! You are welcome to contribute if you like!
Posting a link to AI slop about a mess of AI vibecode is not contributing.
Incidentally, I do not use an IDE, I use vim and emacs.
I do not use Stackoverflow. That is for the third raters.