So... this won't work with llama.cpp right? I need a different runtime for it? I'm new to this LLM stuff.
There are quantizations already. I would suggest running it on KoboldCPP, or any of the UIs recommended on this page: https://huggingface.co/TheBloke/MistralLite-7B-GGUF
Personally, I run the elx2 quantization with exllamav2 because its faster and more coherent in the same amount of vram. You can do that with Aphrodite or Text-gen-ui: https://huggingface.co/models?search=MistralLite%20exl2
<|prompter|>{prompt}</s><|assistant|>
And how would I prompt it with a previous history of the conversation?
<|prompter|>hello</s><|assistant|>
hey there!</s>
introducing the first of its kind, a revolutionary AI...
<|prompter|> is the human message (including the instructon/contexts from my tests) and <|assistant|> is the bot's response.
There are llm wrappers that will save chat histories and reformat them appropriately for different LLMs. For instance, see TUI and its instruction template presets: https://github.com/oobabooga/text-generation-webui/tree/main...
<|prompter|>... prompt goes here</s>
<|assistant|> ... assistant response will go here, maybe followed by </s>
And is my guess about </s> at the end of the assistant thing correct? If so then a conversation might look like this:
<|prompter|>hi</s>
<|assistant|>Hi to you too. What do you want to talk about?</s>
<|prompter|>Tell me facts about snakes</s>
<|assistant|>
Welcome to LLM research land, lol. Mostly there is no documentation, no reproducability, just random hit-and-run releases with some claimed metrics and if you're lucky a model card with the trained prompt.
> Do I need that </s> at the end of the assistant things or not?
...I dunno. I didn't add it in my prompt template, and you can see that the model itself it outputting that token in the responses. Maybe ask the trainers on the HF page.
But TheBloke doesn't (yet) do exl2 quantization. That one is almost better done yourself, as the bpw target is arbitrary, and requires profiling text that should probably fit your intended use (though many just use wikitext as a good default).