Apple was running a model double the size of the available memory. Not sure if that was a sweet spot they found or if you could sacrifice response time to run even bigger models.

The paper is worth a read in full as what they are doing is pretty cool:

https://arxiv.org/pdf/2312.11514

Highlight from the paper...

"Then, we introduce two complementary techniques to minimize data transfer and maximize flash memory throughput:

• Windowing: We load parameters for only the past few tokens, reusing activations from recently computed tokens. This sliding window approach reduces the number of IO requests to load weights.

• Row-column bundling: We store a concatenated row and column of the up-projection and down-projection layers to read bigger contiguous chunks from flash memory. This increases throughput by reading larger chunks."