Technically a wrapper of a wrapper, though llama-cpp-python is quite excellent... TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary).
https://github.com/Mozilla-Ocho/llamafile
You can download any .gguf model (not just the ones in their examples) and run it locally (as long as you have the ram for it). I was running 7B models with ease on an old FX8350 and now 13B models on a 5600X (32GB RAM on both machines).
This wrapper spins up a local web server that runs a simple web frontend to use immediately with no code, but also exposes an OpenAI compatible API for dev work and alt frontends (like SillyTavern).
I see llama.cpp in there, but I didn't go deeper than that. Ollama is not the same thing.
The need to fill more seats on the AI hype train.
(Someone's probably got "release AI software as Google open source" as an OKR)
Very interesting question, as I also read top-level post as saying it's a llama-cpp-python feature, and therefore I could shove some random dolphin-mixtral or whatever I have on my hard drive at it, and it will work.
Functionary is a series of function calling llama architecture finetuned models, not a new base model, if thats what you mean.
Hence, it can be turned into a gguf like any other llama finetune.
It just has a very esoteric prompting format, but llama-cpp-python is specifically set up to leverage it.
> TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary).
I never heard about Functionary, this comment seems to say “I would use llama.cpp plus Functionary”. Now, since llama.cpp allows to plug in several models, I was wondering if Functionary works with several models too, ie is Functionary a library that allows to add function calling to models?
The following answer to my comment helped clarifying that Functionary is not a lib, but a specific model, or that is what I understand from the answer. So the answer is “no, you can’t use functionary with many models (eg it’s not like llama.cpp that works with multiple models) because Functionary is a model itself”
:-)