That being said, I am not sure what’s the killer application for this technology, and without such application adoption is unlikely.
Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
Isn't this how it works today already? Granted you wanted to run it on RAM rather than VRAM.
According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width:
> Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8 GB/s.
And that's just for single 64-bit IC. So way faster and more power efficient.
Consumer platforms have been stuck at dual channel for decades; most of it I attribute to intentional product segmentation. I'm hoping that LLMs might change eventually for an upcoming consumer platforms; going to 4 channel would be really nice.
AI: "Sorry, all the hardware is made for running AI."
Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?
Also, doesn’t it mean that you forgo batching?
How would you do map-reduce across multiple DIMMs w/o extra reads/writes?
PIM implies some sort of distributed compute, which can work for some cases, but I am not sure LLMs are one of them.
Each die-attached PIM accelerator computes online softmax for its own KVs. Then the central unit gathers the softmax intermediates, one intermediate per die, and uses those to compute the final softmax.
The PIM win is that we crater the memory traffic between the central accelerator and the memory dies for attention ops. Most of the attention bandwidth never leaves the memory.
This isn't "run the entire LLM in PIM", no - this is "offload the parts of LLM that benefit from PIM the most to PIM".
You have an eight socket server with 96 memory slots, you add 96x PIM memories into the server (optimistic), load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip.
614 GB/s x 96 = 58,944 GB/s.
Alternatively, the memory is used for embedded inference tasks. You can now upgrade from the limited single or two digit MB SRAM accelerators to reasonably fast single digit gigabyte models. Without MoE you could reach 100 tokens per second with an 8B fp8 model on a single channel. With MoE you might break 500 tokens per second.
Won’t you have a bunch of extra reads/writes via the CPU because these DIMMs won’t be able to compute matrix multiplications?
Build it, and they will come ;)