> M4 Max supports up to 128GB of fast unified memory and up to 546GB/s of memory bandwidth, which is 4x the bandwidth of the latest AI PC chip. This allows developers to easily interact with large language models that have nearly 200 billion parameters.
Having more memory bandwidth is not directly helpful in using larger LLM models. A 200B param model requires at least 200GB RAM quantized down from the original precision (e.g. "bf16") to "q8" (8 bits per parameter), and these laptops don't even have the 200GB RAM that would be required to run inference over that quantized version.
How can you "easily interact with" 200GB of data, in real-time, on a machine with 128GB of memory??