Hacker News Reader

Comment by qwertox | Hacker News Reader

> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch.

It almost hurts, to read that PyTorch is faster.

But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU.

We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

View on HN

Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting around it, the bandwidth that GPUs crave just isn't compatible with modularity.

Even CPUs are starting to move their memory closer to the core in the name of performance, as mentioned Apple is already doing it, Intel is making Xeons with on-chip memory now, and they have a version aimed at consumers on their roadmap.

FYI, most discrete GPUs with discrete memory packages soldered to the board near the GPU are running at substantially higher memory frequencies than the on-package DRAM in Apple's chips. But running GDDR at those speeds costs a lot of power.

I watched a presentation on this today. The presenter focused on the soldering and proximity as well. Is this really the only difference or is this transistor based memory (like L1, L2, etc.)? I get the proximity factor of course (1ft / ns EE rule of thumb). In any case, soldering and proximity don't seem like breakthrough innovations (but maybe I am wrong).

Gpu ram is typically gddr6 or gddr6x which is a different standard to the chips used in ddr5 for example. GPUs have terrible latency to ram, but enormous throughput, and I assume the chips are internally optimized for that. Many aspects of a design change when you choose different latency or clockspeed targets translating into different power / area calculations.

For data rates, as in bandwidth per IO pin, distance is really only a secondary factor. HBM memory, for example, runs at substantially lower data rates than GDDR, yet it sits right next to the GPU die compared to centimeters for the GDDR. And high-speed serial links run at speeds that are an order of magnitude higher than even the internal register files of a CPU.

That's true it's got an impact, but I think there's still space available for "slightly slower with 2x memory" models. For many local uses, new cards are way past the "fast enough" line, but having 64gb on them would be really beneficial.

It's love to see some experiments / different SKUs in this area, given people are already diy-ing extra memory on NVIDIA. (https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-to-... there were stable experiments later on, but I don't have a link now)

Graphics card manufacturers believe that selling high-memory consumer graphics cards will affect the market for commercial computing cards, so they will not do so, that's all.

Nice room for a new player to disrupt then

Problem is, making a board design using an existing GPU chip and sticking more RAM into it is (relatively) simple but of course none of the GPU chip makers would allow partners to do that. Making your own GPU chip that’s competitive with Nvidia or AMD’s current offerings is a massive undertaking and pretty much impossible for a newcomer.

Just look at how much trouble Intel has had breaking into the discrete GPU market or even just how hard it’s been for AMD to compete with Nvidia even with decades of experience in the market.

And if some newcomer could make a competitive GPU with large memory capacity they’d be crazy not to sell it at datacenter prices, maybe just undercutting the others but a few grand but still way more expensive than any consumer GPU you can buy today, even a 4090.

It's not doable at the board level. High-end GPUs use HBM in the same package, connected using a silicon interposer.

Check out PCB back drilling. It's a process where you remove a few hundred microns from the vias that are used to connect GDDR RAMs to the GPUs, to avoid reflections due to the impedance mismatch that's caused by the stub.

When you have a pulse coded signal traveling at close to 10GHz, everything becomes an antenna. The technical problem is that you can't do this with a flimsy connector like the ones used for DIMMs. The reason GDDR can have a bandwidth per pin that is 4 times higher than regular DDR is because they are soldered down on the PCB.

> We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

I imagine it would incur a non trivial latency and cost penalty. The memory modules are placed pretty close to the compute die right now. Cooling would also have to change (the memory modules produce a lot of heat).

But there is also no reason for any of the GPU manufacturers to do this. A skew with twice as much memory can go for a lot more than the difference in memory cost alone

And especially doing "interesting" combinations of gpu and memory.

Like lower end gpu with 16 GB of VRAM, but offering just 8 / 12 GB of VRAM in the middle class and then again 16 GB in the upper class of gpu selection.

I don't disagree but (I know nothing about this btw...) would it not benefit in terms of, say, a L3 cache kind of thing?

Imagine you could stick 2 x 64GB DDR5 DIMMS on the GPU in sockets, would that not be faster to access than the motherboard DIMMS? It won't be as fast as on-die memory of course but could it not act like a sort of halfway house?

> It almost hurts, to read that PyTorch is faster.

Why?

NVIDIA hates that trick.

Reply on news.ycombinator.com