Maybe mention that GGML is under the hood. I see you've forked the library, did you pick up the GQA changes needed for llama2-70b?
Also seems like you are picking quants on people's behalf, which ones?
Also seems like you are picking quants on people's behalf, which ones?
It does use GGML under the hood. Will definitely make sure that's more obvious for compatibility. There's about to be an update as well that includes those recent changes. We do vendor the source code since it made building with Go a lot easier, but definitely don't want to be "forking" it (vs supporting it) – it's a brilliant piece of software!