Hacker News Reader

Requirement:

NVIDIA GeForce™ RTX 30 or 40 Series GPU or NVIDIA RTX™ Ampere or Ada Generation GPU with at least 8GB of VRAM"

Unfortunately the download is taking its time - which kind of base model is it using and what techniques (if any) are they using to offload weights?

Since the demo is 35 GB, my first assumption was it's bundling a ~13B parameter model, but if the requirement is 8 GB VRAM, I assume they're either doing quantization on the user's end or offloading part of the model to the CPU.

(I also hope that Windows 11 is a suggested and not a hard requirement)

For some reason it's actually bundling both LLaMA 13b (24.5GB) and Ministral 7b (13.6GB), but only installed Ministral 7b. I have a 3070ti 8GB, so maybe it installs the other one if you have more VRAM?

I have 3070 and when I choose LLaMA in config it just changes it back to Mistral on launch

8 Gb minimum? So they're excluding the new 3050 6 Gb that is only powered from pcie?

Reply on news.ycombinator.com