Qwen1.5-32B: Fitting the Capstone of the Qwen1.5 Language Model Series

32B (also 34B) is a neat size for prosumer hardware and more compatible with running the llm alongside other apps

context: Qwen 1.5 72B is quite good. It ranks higher than Mistral-Medium (proprietary) on Chabot Arena

edit:

other interesting models in this weight (ha) class:

* Yi-34B (https://huggingface.co/01-ai/Yi-34B)

* Command-R 35B (https://huggingface.co/CohereForAI/c4ai-command-r-v01)

I guess I look forward to seeing 'TheBloke' do a finetune with vram reqs. If my 3090 could fit a good quality quant I'd be interested. I tried to run a mixtral with vram / sys ram offloading and got garbage ouput, but that's probably down to some mistake I made.

Try Ollama or LM Studio. Mixtral and its finetunes work perfectly for me on my RTX3090 with offloading.

GGUF: https://huggingface.co/Qwen/Qwen1.5-32B-Chat-GGUF

AWQ: https://huggingface.co/Qwen/Qwen1.5-32B-Chat-AWQ

Interesting to see a "modern" ~30b sized model - will have to give this a try!

I would like to see how it competes against command-r (non plus)