Hacker News Reader

Comment by wing-_-nuts | Hacker News Reader

I do wonder, say one has 16 gb of vram and 64gb of ram. IIRC mixtral takes ~ 40gb total, if you were using it for creative writing or whatever could you load certain layers into gpu memory and certain layers into system ram? I wonder what the overall speed of this would be?

View on HN

You can partially offload with some backends (e.g. llama.cpp and derivatives) but speed gains from that don't come in until it's mostly offloaded. I have 8GB VRAM and it's not enough to get any boost on mixtral in Q8. 16GB might do better or it might not.

The speed is quite good even on CPU only though, I get 3.5 tokens per second with 6 cores and DDR5-6000. For comparison llama2-70B is less than 1 t/s on the same hardware in Q4. And, subjectively, Mixtral performs better.

I did this with 24GB VRAM and 32GB DDR5, using LM Studio, and it was about as fast as I could read. (I read fast but I'd have to run it again to guess the token rate.)

I'm upgrading to 96GB RAM now to run the larger models, but I do wonder whether it'll be slow when using proportionally less VRAM.

If you are making many requests in batch this works ok because you can shuffle the next layer in while the current one is processing a set of matrix multiplies. This takes it from being a memory bound problem to a flops bound problem. This really only works if you care about throughput and not latency.

I understand that for each token mixtral will only need two (of eight) submodels. I wonder if there is temporal locality and an LRU caching schema could be used.

It is two out of eight at each layer, with 32 layers independent of each other. There are no eight "sub-models".

However, this raises a question: could a slightly more complex router use output layer n-1 to choose experts for layer n+1 (vs n and n+1 today)? This way, there is more time to load the needed experts for the n+1 layer.

With a recent llama.cpp, Mixtral, when run using CPU only (8 threads), produces 4 tokens per second on a Dell Inspiron 7415 laptop with AMD Ryzen 7 5700U, upgraded to 64 GB of RAM. As this is an APU, even one layer is too much for the VRAM, so acceleration cannot be used.

Reply on news.ycombinator.com