Khoj is using the Llama 7B, 4bit quantized, GGML by TheBloke.
It's actually the first offline chat model that gives coherent answers to user queries given notes as context.
And it's interestingly more conversational than GPT3.5+, which is much more formal
Llama 2 gives great answers, even the 7B model. There’s an “uncensored” 7B version as well George Sung has fine-tuned for topics that the default Llama2 model won’t discuss - eg I had trouble having Llama2 review authentication/security code or topics: https://huggingface.co/TheBloke/llama2_7b_chat_uncensored-GG...
From just playing around with it the uncensored model still seems to know where to “draw the line” on sensitive topics but YMMV
If you do end up checking out Ollama you can try it with with this command or there’s an API too (it’s not in the docs yet)
ollama run llama2-uncensoredYeah, I ran into a couple of funny edge cases using Llama v2 with my personal notes. For example, if I ever asked it anything remotely personal (as I would with a personal assistant), it would often start telling me that asking for personal data is unethical. I get it, you have to be careful with the open source LLMs, but still a bit funny. It does work with enough coaxing though.
How are you determining what notes (or snippets of notes?) to be injected as context? Especially given the small 2048 context limit with Llama 1.
We determine note relevance by using cosine similarity between the query and the knowledge base (your note embeddings). We limit the context window for Llama2 to 3 notes (while OpenAI might comfortably take up to 9). The notes are ranked based on most to least similar and truncated based on the context window limit. For the model we're using, we're still limited to 2048 tokens for Llama v2.
The 7B version was a decent enough starting point in terms of what it can answer (and way fewer folks can run a 13B on their machine).
If you really want you can just replace the 7B model file with the 13B one under the ~/.cache/gpt4all directory on your device and it should just work.