Processing in Memory: DRAM Is About to Do Math

There's different kinds of RAM. CPUs are using SRAM for their caches (which usually have ~6 transistors per bit cell, IIRC, although I'm not a hardware guy, so take this stuff with a grain of salt). Your RAM sticks are using DRAM, which are 1 transistor and one capacitor per bit cell.

Now, this is where my understanding of the actual physical issues really gets sketchy, but the processes that are good at making the CPUs just aren't good at making DRAM, and vice versa. (I would guess that it has specifically to do with the capacitor, but that's purely a guess.) SRAM isn't a problem for the CPU processes, because SRAM cells are closer to the other kind of logic you set up in CPUs.

The problem with your rambling and speculation is that you're not making any attempt to think about what conditions would be necessary for a technology to succeed in the market and gain widespread adoption across many market segments. I stand by my assertion that it is impossible for compute-in-memory chips to entirely replace commodity DRAM, especially in any market segment where the compute portion of those memory chips is not critically valuable. Any remotely plausible, useful compute-in-memory design will have significantly lower capacity and higher cost than pure commodity DRAM; this is not a low-overhead added feature. Any market segment where the compute capabilities of compute-in-memory were so marginally valuable that a non-trivial number of the customers wanted to disable the compute is a market segment where everybody would still be buying mostly ordinary DRAM.

A CPU is mostly transistors, so SRAM is made of the same “stuff” as the CPU logic itself.

This makes it easy to put SRAM on the same chip as a CPU, but difficult to put dense amounts of DRAM on that same chip. You could etch CPU and DRAM with the same process but both would suffer in density and therefore speed, cost and power efficiency. That’s the fundamental reason why DRAM and CPU/GPU are on different chips with a fast interconnect between them. This is so even in devices like the iPhone where one designer controls and designs the whole thing and could, in theory, fab a single design with both. But they don’t.

(There are some old exceptions to this like the PS2 shipping with a small amount of on-die DRAM. And process incompatibility is not the only reason for keeping the two chips separate. But if the processes weren’t incompatible there would certainly be high-end fast CPUs/GPUs with big gobs of DRAM.)

The drastically different scaling trends for logic and SRAM on the same die are important, but I'm not sure how relevant it is to LLM inference. Cerebras has already shown that the amount of SRAM you can get out of a wafer is an absolute pittance compared to what frontier LLMs require; a multi-million dollar machine with ~40GB of SRAM is not a good solution for big LLMs no matter how fast it may be for small models. Denser forms of memory are almost mandatory for doing LLM inference on the largest models, and it seems pretty clear that the most cost-effective architectures we can build today for LLM inference would not be dominated by SRAM. You'll still have SRAM caches, but it doesn't make sense as the primary storage for the entire model's weights.