Hacker News Reader

I think the first Rust ML framework to gain massive traction will be the one that supports quantisation out of the box.

At the moment there's no Rust ML library that does what llama.cpp or ggml can do.

Nowhere near as neat as candle or ggml, but just released a 4-bit rust llama2 implementation with simd. Runs pretty fast.

https://github.com/srush/llama2.rs/

That's really cool.

That is _really_ cool. Would be interested in some requirements being posted. Eg RAM required for the 70B model if CPU, VRAM required if GPU, etc.

edit: I know it's memory mapping, but that still loads data in RAM to execute so if you had 512MiB of RAM it would likely be slow as hell.

Yeah, it's CPU only, and it is using about 38g for 70B and 7g 7B. Guessing that is mostly from the large caches that it keeps around for efficiency. If you wanted to pay some computational cost, you could likely get that down by quantizing activations. Llama2 has some of these tricks built in automatically, for instance grouped query attention.

They already support quantization out of the box!

https://github.com/huggingface/candle/pull/314/files

Reply on news.ycombinator.com