I saw this on HN before, but I thought it was another from-scratch llama implementation... Which is fine, but much less interesting to me, as a from-scratch implementation probably not as fast/feature packed as llama.cpp or the TVM implementation.
Keeping up with llama.cpp's rapid evolution is very difficult, and there's a need for projects like this.
* https://github.com/jmorganca/ollama/blob/main/docs/api.md
* https://github.com/ggerganov/llama.cpp/blob/master/examples/...
Some like c0sogi/llama-api are pretty neat because they support concurrency, and supports multiple backends (llama.cpp and Exllama, although it could be expanded).
While you might lose out on some low-level configurability, being able to easily swap between OpenAI and local models is a big win in my book.
To me a killer feature would be easily running different models simultaneously such as one for embeddings and another for completion (e.g. Chat). This likely can be done already by specifying the model parameter in Ollama (and others) but I've not explored it much yet.
https://langroid.github.io/langroid/blog/2023/09/19/language...
- https://github.com/openai/triton
- https://github.com/NVIDIA/TensorRT
TVM and other compiler-based approaches seem to really perform really well and make supporting different backends really easy. A good friend who's been in this space for a while told me llama.cpp is sort of a "hand crafted" version of what these compilers could output, which I think speaks to the craftmanship Georgi and the ggml team have put into llama.cpp, but also the opportunity to "compile" versions of llama.cpp for other model architectures or platforms.
Its very fast on Vulkan, and from what I understand fast on metal, but its not as feature packed as llama.cpp yet.