Hacker News Reader

Comment by MBCook | Hacker News Reader

I wonder how much of the model you can avoid loading before you start to see a real performance difference.

Let’s say you want to maintain 90% of everything-in-RAM performance.

Can you get away with only using half the memory? Do you need 90% of the memory? Maybe 95%?

Basically how fast do you lose performance compared to the maximum by cutting RAM. The charts are comparing their algorithm vs a basic one for the less RAM case, which is a different (but good!) question.

If you can get good performance by NOT loading an entire eight gig model into memory of a cell phone that’s obviously a very useful thing.

View on HN

Apple was running a model double the size of the available memory. Not sure if that was a sweet spot they found or if you could sacrifice response time to run even bigger models.

The paper is worth a read in full as what they are doing is pretty cool:

https://arxiv.org/pdf/2312.11514

Highlight from the paper...

"Then, we introduce two complementary techniques to minimize data transfer and maximize flash memory throughput:

• Windowing: We load parameters for only the past few tokens, reusing activations from recently computed tokens. This sliding window approach reduces the number of IO requests to load weights.

• Row-column bundling: We store a concatenated row and column of the up-projection and down-projection layers to read bigger contiguous chunks from flash memory. This increases throughput by reading larger chunks."

I was thinking a couple of days ago about this concept of windowing for LLMs, but I lack the technical skills to implement it. Now Apple just published a paper on it. This is what I call synchronicity

Is this https://ipads.se.sjtu.edu.cn/_media/publications/powerinfer-... related? Could it be the origin?

By the time we put our paper on arxiv this paper was not out so we were not aware of, but it is similar in some ways. Both this paper and us are reliant on our previous paper https://arxiv.org/abs/2310.04564 and dejavu https://arxiv.org/abs/2310.17157.

They are targeting limited gpu memory and limit cpu to gpu memory transfer. I don't know how it could be useful on Macs because MacBooks have a unified memory and you don't need to do that transfer necessarily.

Half the memory was just an example not a sweet spot. With smaller window sizes you can use less memory, but it will come with the cost of loading more from flash.

p.s: The window size in the paper is showing how many token's feed forward layer is in the memory.

I'm just thinking out loud. Nothing in this post is authoritative.

Theoretically, the time consumed to inference a single token with part of the model stored in flash should be equal to the time to inference that token if the whole model was in RAM, plus the time required to load the part of the model stored in flash memory.

I assume we do not need to write back to flash, but I'm not an LLM expert so I could be wrong.

I assume we have many (more than 10) layers so we can leave a fairly small amount of our RAM available to load one layer after another. Most nontrivial LLMs have many dozens of layers, so this seems plausible.

If we are not bottlenecking on our RAM during inferencing then we might be able to load the next layer from flash into RAM through a DMA transfer while inferencing our current layer. I don't think that would work on single-processor systems due to us always bottlenecking on RAM.

Maybe a dual-processor system could load one layer into RAM on one processor while inferencing on the previous layer on the other processor, and thus run really big LLMs in a small amount of RAM?

I'm sitting next to a pile of parts to build a new LLM AI machine. (z840, dual processor) and I look forward to playing with this stuff.

There was also a throwaway comment about experts:

"Mixture of Experts (Yiet al., 2023) have a sparse structure in their feed forward layer. This property can be used to combine with our method for enabling larger MoEs on device."

Assuming this implementation would allow for running Mixtral 8x7b on a 16Gb M1, I'm happy.

I would think so, I have it quantized at 8 bit (q8) and that ticks in at ~47GB.

Q4 should be well below the 32GB (2x 16GB).

I am wondering if the same implementation could be done for RAM to CPU, too. Those are said to be data transfer limited, so minimizing the RAM to CPU cache transfer should help there, too?

Q4 comes out to be ~26GB but Apple doesn't let you load it on a 32GB Mac machine because they put a limit on the max usable unified memory at ~21GB (`device. recommendedMaxWorkingSetSize`) [1]. So for Q4 Mixtral MoE you'd need a 64GB Mac machine unfortunately.

Unless you use this hack [2].

[1] https://developer.apple.com/forums/thread/732035

[2] https://github.com/ggerganov/llama.cpp/discussions/2182#disc...

sysctls aren't a hack exactly, it's there so you can change it.

As for why it's not the default, it's mostly because wiring all your memory will crash the computer pretty fast.

There’s a brand new hybrid quantization for Mixtral out that uses 4b for shared neurons and 2b for experts, which does not bleed much perplexity, but fits it into a 32G machine. Haven’t had it in hand yet and no link here on mobile, but can’t wait to try.

Reply on news.ycombinator.com