Hacker News Reader

I feel like this is the greatest demand for LLMs at the moment too.

It's hard to believe we're only 8 months into this industry, so I imagine we'll start seeing smaller footprints soon.

8 months from what point?

Gpt3 is 36 months old. Dalle-e is 28 months old. Even StableDiffusion is like 11 months old.

Fair, I should have said 8 months since the market exploded.

We already do. MLC-LLM and Llama.cpp have Vulkan/OpenCL/Metal 3 bit implementations. That can run llama 7B (or maybe even 13b?) in 8GB.

TBH devices just need more ram for coherent output though. Llama 13b and 33b are so much "smarter" and more coherent than 7B with 3 bit quant.

13b q5 llama report „total VRAM used: 8321 MB” so 3bit will most likely fit into 8GB.