GPU's are already closer to "in-memory compute" compared to CPU's. It's just taking the existing pattern of NUMA (non-uniform memory access) to a greater extent, as a principled approach to the so-called 'Von Neumann bottleneck'.

In-memory compute is very easy, you just don't have to fall for the pipe dream of using the same process for both the memory and the compute. All you have to do is follow a package on package strategy like we already do with smartphones. A Raspberry PI 5 gets 25GB/s memory bandwidth and it only has a single DRAM chip if I recall correctly.

So if you had a DIMM with 16 of these chips, you would already be on the same bandwidth as HBM. 96 DIMMs and you get 40TB/s memory bandwidth.

Can you explain further why you believe it isn't worth exploring? Is it that you can't imagine how it would scale to the level of today's high performance compute hardware?

Just ten years back, squeezing ML models onto microcontrollers sounded completely insane, given their tight memory and power constraints. We've seen NN compilers developed, game-changing techniques like quantization, pruning, and graph-level optimizations pruning. This allowed deployment of ML models in microcontrollers with a newly developed framework like TFLite Micro.