The Memory Wall: Past, Present, and Future of DRAM | Hacker News Reader

The Memory Wall: Past, Present, and Future of DRAM

semianalysis.com

106 pointsklelatti27 comments

Back when I was young, which isn't that far ago, one tried to put as much pre-computed stuff into memory as it was much faster than the CPU. Lookup tables left, right and center.

These days you can do thousands of calculations waiting for a few bytes of memory. And not only is the speed difference getting worse, but memory sizes aren't keeping up.

Guess we're not far away from compressing stuff before putting it in memory is something you'd want to do most of the time. LZ4 decompression[1] is already just a factor of a few away from memcpy speed.

[1]: https://github.com/lz4/lz4

Reminded of Kaze Emanuars work on optimising N64 software. (https://www.youtube.com/channel/UCuvSqzfO_LV_QzHdmEj84SQ)

The RAM was so terrible that essentially you try to keep the processors running in cache for as long as possible. RAM access is painful.

There is a performance profiling tool built into F3DEX3 that now shows that approximately 70% of the time the system is idle while running Zelda OoT. It is just waiting for memory transfers. The folks at SGI/RAMBUS cut corners a little too hard building that system.

But turns out this kind of performance profile is just prep for were we are heading apparently.

I was re-reminded of this when watching a performance analysis video that may or may not have been posted here (sometimes I get things here, or reddit, but sometimes the real story is in the related videos). It doesn't take a very big lookup table for it to be faster to rerun the calculations.

Especially when you throw multiprocessing in. We need better benchmarking tools that load up competing workloads in the background so you can tell how your optimization really works in production instead of in your little toy universe in the benchmark.

fixed function compression on-die would be interesting if applied to this

On the latter point, macOS had had compression memory for a long time by now and some Linux distributions also use it out of the box (I don’t know anything about Windows).

One of the time series databases streams compressed blocks in memory when doing searches, doing distinct blocks per core. For some scenarios it's faster to do a table scan than keep extra indexes hot.

That's compression for currently inactive pages, not nearly as aggressive as compressing as a matter of course.

It also has Compressed Memory since a version of Windows 10. You can see it in Task Manager.

Even in the 90's I recall decompressing ZIP files on a 486 being limited by the HDD speed. It felt like even then we would head towards compressed memory systems once it could be compressed quickly enough.

I think the grandparent was talking about transparent memory bus data compression. CPU would fetch some bag of bytes and decompress to cache.

This is basically intended to be fast swap rather than something to be applied to active pages.

I think quantization for large language models already do something like this - they compress the parameters in memory and then decompress when performing the forward passes

I have been stating this for at least 12+ years now. On HN and in many places. And it is still not common knowledge. Hopefully this article get spread and reposted many times in the future.

DRAM isn't getting (much) cheaper. And it will continue to be the case in next 10 years. Considering there is nothing on the roadmap to suggest any breakthrough.

People old enough may remember in the late 00s and even up to mid 10s, there are words like we will get 16GB computer as baseline, or I could work with 64GB soon.

Reality is that DRAM price has fluctuate lately within the same range in the past ~13 years. It was only in 2023 the price floor broke through the $2/GB barrier.

Unless we somehow found a way that could magically shrink the capacitor by a substantial amount. Or we change the way we do programming.

On the other hand, my home workstation has 128GB of memory, for something like $300. That is so much capacity 99% of the time I no longer have to even consider doing something stupid on a data processing task, it just works.

It is too early for me to imagine what capabilities are unlocked if I personally had say 1TB of memory, but large RAM capacities seem only required for mammoth database servers.

It was the latest console generation that merely doubled the RAM from the previous generation that showed this in action. That plus silly things like Macbooks with 8GB base line for over a decade.

I'm curious if it is possible to design an AI accelerator built around ROM that is programmable via maskless lithography. The idea being that 90% of the chip is being produced using conventional mask based semiconductor technology, but the data is written through electron beam lithography.

Given a regular ROM structure on the silicon wafer, it would be possible to take advantage of parallel beams and design an electron beam lithography machine specifically for ROM to reduce the cost of programming. The final frontier would be to build a "wafer scale engine"-esque single wafer chip and we would have essentially reached the limits of what is achievable through clever design of silicon based semiconductors.

Compute in memory is exciting, but it’s going to be hard to pull off. Transistors go underneath capacitors, which means they have to survive the subsequent fabrication of the capacitors without thermal processing disrupting their carefully engineered dopant profiles. It’s not just conservatism that keeps on die logic simple. Bonding, as suggested by the article, is probably the only option, but that brings some absolutely hilarious alignment issues along with it. I’m sure it’s coming, but it won’t be cheap.

SK Hynix has already developed PIM and nobody cares.

https://news.skhynix.com/sk-hynix-develops-pim-next-generati...

Can someone explain like I’m five why DRAM latency basically hasn’t improved in decades? I remember 60ns FPM SIMMs in our 486 back in the 1990s. Doesn’t seem any faster now.

What happened to SRAM ?

Still in wide use in CPU caches. It does not scale well to smaller sizes, and inherently requires more transistors than DRAM, which means it's not cost-effective for main memory.

I forgot were I saw it but it was an analysis of cache to die size and how it has effectively stopped scaling. Larger cache just means larger chips now.

It sort of makes sense because SRAM cannot have the same transient nature of a calculations transistor. It has to hold the state for longer than one 3 billionth of a second. So it has to be a little more robust.

This is my intuitive take so it could be completely wrong.

The densest kind of SRAM has stopped scaling. However, it's instructive to compare the size of the L3 on AMD Zen4 and Zen5 dies. There is still at least some scaling to be had for SRAM that's built to be fast.

(Of course, this is still very irrelevant for comparison to DRAM; the price difference is huge.)

Probably on Asianometry: https://www.youtube.com/watch?v=2G4_RZo41Zw

It's huge compared to DRAM. It's worth making a few megabytes of SRAM per core, but not a lot more than that.

if Sally Mckee, who coined the term "Memory Wall", had a nickel for each time it gets mentioned, she'd have a lot of nickels

I think that 'memory wall' is completely different from 'memory hole', but I forget how.

Reply on news.ycombinator.com