Hacker News Reader

The FP16 7B version runs on my Ubuntu XPS with 32GB memory, ~300ms/token. 13B also works but results aren't really good (the model will loop after a few sentences) so parameters probably need tuning.

So far I'm unable to reliably generate outputs in a different language than English, the model will very quickly start to translate (even if it's not asked) or just switch to English.

The mac M1/M2 shares CPU and GPU memory. Not all GPUs have enough memory available. Transferring data to/from the GPU has a perf disadvantage. Hence, few PCs are capable of running LLMs.

The article and linked GitHub project both say this runs on the CPU. The GitHub project notes the quantization doesn’t work right with other systems for some reason but if you have enough RAM to quantize it you have enough RAM to run it without quantizing it anyways.

Aha. Then I guess performance is the only issue. I guess it should run fine on Intel.

This is a Mac thread. Virtually everyone is using a PC. I test between a HEDT Linux system and a recent Intel build on Windows 11.

Took me around 8 hours to build and deploy 4bit. 7B and 13B worked great, still working on the quantized 30B weights.