Hacker News Reader

Comment by jsheard | Hacker News Reader

Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting around it, the bandwidth that GPUs crave just isn't compatible with modularity.

Even CPUs are starting to move their memory closer to the core in the name of performance, as mentioned Apple is already doing it, Intel is making Xeons with on-chip memory now, and they have a version aimed at consumers on their roadmap.

View on HN

That's true it's got an impact, but I think there's still space available for "slightly slower with 2x memory" models. For many local uses, new cards are way past the "fast enough" line, but having 64gb on them would be really beneficial.

It's love to see some experiments / different SKUs in this area, given people are already diy-ing extra memory on NVIDIA. (https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-to-... there were stable experiments later on, but I don't have a link now)

Graphics card manufacturers believe that selling high-memory consumer graphics cards will affect the market for commercial computing cards, so they will not do so, that's all.

Nice room for a new player to disrupt then

Problem is, making a board design using an existing GPU chip and sticking more RAM into it is (relatively) simple but of course none of the GPU chip makers would allow partners to do that. Making your own GPU chip that’s competitive with Nvidia or AMD’s current offerings is a massive undertaking and pretty much impossible for a newcomer.

Just look at how much trouble Intel has had breaking into the discrete GPU market or even just how hard it’s been for AMD to compete with Nvidia even with decades of experience in the market.

And if some newcomer could make a competitive GPU with large memory capacity they’d be crazy not to sell it at datacenter prices, maybe just undercutting the others but a few grand but still way more expensive than any consumer GPU you can buy today, even a 4090.

It's not doable at the board level. High-end GPUs use HBM in the same package, connected using a silicon interposer.

FYI, most discrete GPUs with discrete memory packages soldered to the board near the GPU are running at substantially higher memory frequencies than the on-package DRAM in Apple's chips. But running GDDR at those speeds costs a lot of power.

I watched a presentation on this today. The presenter focused on the soldering and proximity as well. Is this really the only difference or is this transistor based memory (like L1, L2, etc.)? I get the proximity factor of course (1ft / ns EE rule of thumb). In any case, soldering and proximity don't seem like breakthrough innovations (but maybe I am wrong).

Gpu ram is typically gddr6 or gddr6x which is a different standard to the chips used in ddr5 for example. GPUs have terrible latency to ram, but enormous throughput, and I assume the chips are internally optimized for that. Many aspects of a design change when you choose different latency or clockspeed targets translating into different power / area calculations.

For data rates, as in bandwidth per IO pin, distance is really only a secondary factor. HBM memory, for example, runs at substantially lower data rates than GDDR, yet it sits right next to the GPU die compared to centimeters for the GDDR. And high-speed serial links run at speeds that are an order of magnitude higher than even the internal register files of a CPU.

Reply on news.ycombinator.com