Hacker News Reader

AI inference is basically 1 operation iterated over every single model weight. That's what's makes computer in memory attractive and feasible.

General purpose workloads would require much more silicon real estate and are not (typically) nearly as bandwidth hungry. Just like not all CPU workloads benefit from being adapted to a GPU, even fewer will benefit from being adapted to processing in RAM.

View on HN

AI is not intrinsically matmul. Current models are a framing of the problem as matmul to enable the bulk operations necessary. There are many calculations that can (and have been) reframed into matrix form to achieve similar results. More than once I have seen a matmul version of something that I thought couldn't have been representable that way. I'm not sure if the greater benefit is the performance or the ability to conceptually manage the problem in you head.

Not all algorithms can be converted of course, but a lot of that is because there are some inherently unparallelizable problems.

Many algorithms can be reasonably converted into matmul. Not many algorithms can be reasonably converted into very low precision matmul. And that's all that these things are going to do.

Reply on news.ycombinator.com