Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?
Also, doesn’t it mean that you forgo batching?
How would you do map-reduce across multiple DIMMs w/o extra reads/writes?
PIM implies some sort of distributed compute, which can work for some cases, but I am not sure LLMs are one of them.
Each die-attached PIM accelerator computes online softmax for its own KVs. Then the central unit gathers the softmax intermediates, one intermediate per die, and uses those to compute the final softmax.
The PIM win is that we crater the memory traffic between the central accelerator and the memory dies for attention ops. Most of the attention bandwidth never leaves the memory.
This isn't "run the entire LLM in PIM", no - this is "offload the parts of LLM that benefit from PIM the most to PIM".
Isn't this how it works today already? Granted you wanted to run it on RAM rather than VRAM.
According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width:
> Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8 GB/s.
And that's just for single 64-bit IC. So way faster and more power efficient.
Consumer platforms have been stuck at dual channel for decades; most of it I attribute to intentional product segmentation. I'm hoping that LLMs might change eventually for an upcoming consumer platforms; going to 4 channel would be really nice.
AI: "Sorry, all the hardware is made for running AI."