There's actually already a fork that implements the preliminaries:

https://github.com/ggml-org/llama.cpp/discussions/21961