Hacker News Reader

Comment by londons_explore | Hacker News Reader

Whilst processing in memory is clearly the future, I am unconvinced by this implementation.

Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2).

To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a 'around the chip' ring shift register to pass every element of one matrix past every element of the other.

View on HN

"Movement is the main thing" is precisely why pursuing compute-in-RAM makes some sort of sense to begin with. But DRAM fabrication processes are quite specialized and do not perform well with pure compute logic. The overall profile of this thing will arguably be similar to a rather weak NPU, though with much better memory bandwidth - one key limitation, as with NPUs, will be the bespoke programming model and lack of support for the latest compressed/quantized number formats, which heavily limits the usefulness of being able to access memory directly. GPUs, even weak iGPUs, can dequantize/pad parameters on the fly which adds a lot of flexibility - and expose standard, well understood compute capabilities via CUDA, Metal or Vulkan. This is not quite comparable unfortunately.

I was under the impression they're doing a chiplet thing to get heterogenous processes for that reason.

If what we want to do with this is make cheap QKV sweeps, then "a weak NPU with a lot of mem bandwidth" seems good enough? Exactly the tool for that job, and nothing else.

Also spares us the trouble of dealing with weights. By the time we're in QKV realm, the weights have already weighted.

Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.

Exactly where I see this going as well. Sure, a smartphone might be a nice place to introduce such tech. But matrix multiplication is literally where all non-labour jobs are going - this is the bedrock for efficient (time, energy) machine learning and inference.

If AI really is going to eat all our jobs, then matrix multiplication in memory is almost a requirement.

Turning RAM into an active coprocessor does feel like an awkward model for that though.

Intel's Advanced Matrix Extensions (AMX) and the dedicated tile registers, which are basically systolic arrays IIUC, seem like a better fit.

But Samsung doesn't have that option so they're playing the card they have.

I think useful flops per dollar will outweigh any "awkwardness" factor.

Elegant designs don't win anymore.

With the size of these matrices I don't think they are even meaningfully colocated with themselves in memory. You'll end up with some dataflow TPU architecture anyway because you'll have to stream the second matrix to multiply against.

People have been calling processing-in-memory "the future" since at least the 1980s. No one has been able to reduce the concept to a useful implementation but there is a long history of failed attempts.

At this point processing-in-memory has taken on the aura of fusion power.

OLED also has a deep history with a pretty famous opinions that it is impossible and a waste of time and money for companies to invest in the development. So does AI. From inception to product it is normal for decades to pass.

Compute needs change. AI needs are pretty unique in terms of scale and type of compute requirements to anything else so far.

I think Wright brothers had long history of failed attempts.

And all of sudden, no passengers are crossing Atlantic on ship.

Well, when you fail at everything, you want other's to fail too. Luckily for us, your failure doesn't affect others.

> Whilst processing in memory is clearly the future

How clear is that? The idea has been around for about 60 years, and many attempts made by people who thought the same thing. Maybe this time it'll be the future.

Because now the usecase fits the needs.

Maybe. The use cases have always been around FP arithmetic over arrays, because that's what is easy to parallelize. I saw a standalone systolic array box attached to a MicroVAX circa 1990. Lots of PIM approaches in the mid-90s, too, but mostly what survived from that era are shared memory NUMA multiprocessors and using GPUs for general purpose computing.

This is really just restating the assertion I'm asking about. Previous work also thought their use cases fit the needs.

Everything changes all the time in computing, so there are lots of differences. I'm not asking if things are different or claiming it won't work this time either. I'm asking what it is this time that makes it different / obvious. Not rhetorical, I'm interested if someone can actually explain. Preferably with trends and numbers that have $ and flops and picojoules and gates and bits in their units.

you're absolutely correct that pim without a real discussion about how that works in a broader communications context is kind of useless.

what I find strange is the adoption of a standard synchronous dram interface. that's a horrible left over piece of architecture that severely constrains the applicability of this device. control flow on the dram side can't initiate any transactions on its own, or respond after work has been done - its like usb, except with a hard limit on the response.

that severely limits the utility of the in-memory processors to doing things like encryption and compression - but even then those impose delays that effect the consistency model across that interface.

Not if you have duplicates of rows on the first matrix, which can be done very efficiently if you build specialized hardware. Then its all just forward in parallel.

Reply on news.ycombinator.com