Comment by bob1029 | Hacker News Reader

Comment by bob1029 | Hacker News Reader

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applications noted above eventually got their own flavors.

I think the Von Neumann bottleneck is mostly a feature. The fact that communication of information across distances is expensive should not be immediately assumed to mean that it is universally flawed to do this. You are paying for something when you use all those joules. I'd argue we are usually wasting our energy with regard to information communication (e.g., lighting up a network interface & copper because we couldn't be bothered to use SQLite), but other times this stuff is fundamentally required for practical solutions to exist.

I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module.

Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region.

Let's say you would need to run a sum over megabytes of data like for accounting. You could just ask RAM to do it and load just the result. The bandwidth would could be 8x higher and software could stay the same.

Doing scalar operations, frequent dereferencing and similar would not get much peformance benefit in many cases, as loading and accessing CPU cache is often much faster. But simple vector operations over large data could be massive.

Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.

Personally I'm a big fan of the "in-ram accelerator" idea especially for server space. Doing fuzzy search in RAM could be massive performance improvement.

> Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.

I think we are essentially reinventing SSE, AVX & friends from first principles. This is already being done. Compare the speed of libjpegturbo to a non-vectorized implementation and you'll find a 2-4x difference in throughput.

Isn't any external accelerator just reinventing SSE and AVX? CPU is a general purpose processor it could do GPU's job but we don't force it to, because it's faster to do it on external accelerator.

Yeah this sounds a lot like a natural evolution of SIMD for me, just cut out the middleman and put the SIMD units straight into RAM.

I can imagine some power savings for always-on display applications too. Rather than periodically waking the CPU/GPU to update the frame buffer, you can just stash small bits of periodic logic in memory (e.g. move the second hand of a clock).

I'm having a hard time imagining this SIMD replacement except for extremely narrow use cases. Are you suggesting the PIM would have a full blown IO controller and cache subsystem to fetch remote operands?

I assume PIM is only going to work well for chunky streaming over the data within that particular memory module. Something that address and operate on whole rows at once and has minimal buffering between the RAM access ports and the PIM register state.

A lot of SIMD code can be on two array operands, and I expect this PIM approach only works well if both are stored locally in the same memory "local" to the PIM and where it can efficiently interleave at the natural addresisng and access widths. Too much random access or needing "remote" data sounds like the point where PIM fails and you need the elaborate memory IO controllers and caching subsystems of CPUs sitting on top of the distributed memory modules..?

> PIM would have a full blown IO controller and cache subsystem to fetch remote operands

Don't we already have those in mainstream computing in the form of dedicated silicon in DMA controllers? Programmed input–output performance is often low throughput, high jitter and uses a lot of CPU.

Yes, but a DMA controller sit out on the memory bus to do the same kind of work the CPU could be doing, controlling bus traffic between memory modules.

I think the whole idea of PIM is to be able to scale up and do work locally within the memory module, without the bottleneck of the system memory bus. This only works for embarrassingly parallel workloads that don't actually depend on the bisection bandwidth across the global memory.

If you start talking about PIM that is all interconnected, your application is back to being bound by the system memory bus. Maybe it's a new package design, but you're basically building yet another NUMA interconnect architecture, not a PIM architecture.

If Infiniband does this for MPI on the network and realizes Sun's "The network is the computer" dream, I believe we can do this for other parts of the hardware, as well. Not only for AI, HPC will love this idea.

> run a sum over megabytes of data like for accounting

It's been many decades since the last time somebody ran a sum over megabytes of data for accounting and though "damn that's a bottleneck I need to optimize".

> Personally I'm a big fan of the "in-ram accelerator" idea especially for server space.

The operations this model supports are so extremely limited that you would be hard pressed to find applications where it's worth it.

Well I do. I run blockchain analysis. The RAM is very much a bottleneck in many cases even with DDR5.

Data movement and local operations are still bottlenecked today on memory bandwidth. Butterfly primitives, sorting/fft/1D-convolution, the whole cub library, could be ported there and have great performance wins. But the pain of programming and maintaining code using this...

Yes, and no.

Two decades ago it was a challenge to get people to see that what they were doing was preventing horizontal scaling. Today horizontal scaling is table stakes and people don't even always register that they are doing it. It's just how we do things, no thoughts.

PIM requires problems to be decomposed into horizontal scaling problems. Then what you should do with PIM is take a problem that used to be solved by 2 racks of computers and squeeze it down to less than half a rack by stuffing a bunch of these into a single box to do 8-10x as much work per box (and double the cluster size to offset Jevons' Paradox because it's so cheap now that you'll do 2x as much of it)

In a world where AI is writing all of the code, the difficulty of the task may no longer be a blocker.

Indeed. SIMD type code used to be too hard for me to write very often, but now it’s easy to crank out tons of it.

There are fundamental issues here and I think the article only touched on a few. On the software side this completely blows up the whole virtual memory concept. We will need different operating systems.

why would it? the parent OS can already handle physically contiguous allocations so these should be no different (with the exception that a separate interface can be used to do compute over these buffers/pages).

If it requires physically contiguous RAM to work, then it's not really participating in the full virtual memory system, really. It would be using an exception to it, that can be accommodated to some extent by the OS, but not sitting in demand-paged storage like the rest of the system.

And the security folks will have tons of fun with this enormously leaky abstraction too.

Hey, just don't store your pagetables in PIM capable sections :)

Linux has been dealing with this kind of thing for over a decade now. Specialized SoCs love their memory carveouts.

You could still do map-reduce operations, but for it to really fly, what you'd want is a side channel between the memory chips that allows the reduce to happen out of band from the front-side bus and the reduction to be sent to the CPU. And any workflow where you can stream the reduction to the CPU that would be even better for latency.

That really doesn't make sense because your system is already using memory that needs to be physically contiguous but mapped virtually. Also, would you say pinned memory is not part of the virtual memory system??

You're only restricted by the fragmentation of the system memory which is an issue yes, but it's dealt with in other ways.

Maybe PIM will push this forward, but I still think we're doing something fundamentally wrong by not just embracing NUMA and trying to do something Sun tried decades ago, which is have number of cores share 4GB of semiprivate working memory.

We've kind of half-assed it with DDR memory banks, but it mostly introduces mysterious slowdowns that are difficult to reason about and I think we would be better served I think by making a formal thing. Instead of introducing an L4 cache we could do this instead, and reduce the size of the L1-L3 caches, which shortens lookup time and thus latency.

For legacy apps, you could provide facilities for the OS to 'page' blocks in from main memory, but the speed would come from managing the workload imperatively, starting loads in the background before the data is actually needed, and dumps after it is last touched.

Intel SCC tried the extreme version of that. Though you could argue their Message Passing Buffers weren't shaped quite right (too small, hard for OS to use across trust domains). https://en.wikipedia.org/wiki/Single-chip_Cloud_Computer

You could also take a half-step in that direction with the DEC Alpha processor's extremely relaxed memory consistency model, and see how well things program for that. When I had a chance to poke at them, Alphas were about 2-3x faster than Intels, with much less ecosystem optimization effort burned -- and most software worked just fine.

One thing this makes possible – accelerated memcpy. Of course, it is only worthwhile once the memory buffer to be copied is large enough. But the copy speed could be greatly increased if it could all be done inside the memory module. And of course, that only works if the source and destination buffers are physically on the same module. But, if your application allocates 1GB pages, the OS could attempt to ensure they are on the same memory module.

Accelerated memcpy is already offered by some Intel server systems (QuickData aka IOATDMA aka CBDMA aka SDMA), but it uses a memory-to-memory DMA engine on the CPU die, so the memory to be copied still has to travel back and forward between the CPU and the memory modules via the memory controller, even though using a DMA engine means it isn't consuming a CPU core. With this, the same process could be made much faster, bypassing the CPU entirely, provided the source and destination are on the same module.

Reply on news.ycombinator.com