Comment by jesprenj | Hacker News Reader

Comment by jesprenj | Hacker News Reader

> Although the GPU performs inference faster than the FPGA, one of the primary bottlenecks of deep learning inference is memory bandwidth and the availability of on-chip memory (Balasubramanian et al., 2021). A RTX 3090 has 24GB VRAM running at 1219 MHz with a base core clock of 1395 MHz (TechPowerUp, 2024). In comparison, a VU9P FPGA has 345.9 MB of combined on-chip BRAM and URAM, running at a much slower clock speed of around 200-300 MHz depending on the module; however, with much lower clock speeds, the FPGA is able to achieve better efficiency on power and energy consumption, as shown below.

So as far as I can understand, the biggest "bottleneck"/limiting factor with using FPGAs for LLMs is the available memory -- with current large models exceeding 40 GiB in parameter size, GPUs and TPUs with DRAM look like the only way to go forward for the months to come ... Thoughts?

Wouldn't be surprised if AMD or Intel come up with an FPGA especially for this application. Atleast AMD advertises a lot with their AI FPGA stuff, so they'll probably build one with either a lot more BRAM or the ability to attach some very fast RAM? But going from a few Megabytes of RAM to Gigabytes sounds very expensive. DRAM is just too slow, I guess

Check out the Agilex-M device: https://www.intel.com/content/www/us/en/products/details/fpg...

Up to 38 TFLOPs of FP16 performance, up to 116 Gbps transceiver rates, and up to 3.9M logical elements (LE).

Memory bandwidth of over 1TBps using the NoC, using in-package HBM2E (up to 32GB capacity) and hardened DDR5/LPDDR5 memory controller (supporting 5,600 Mbps).

Maybe doesn't come into the power envelope (Agilex 5 is looking good on that front) of a low-power device, but it's an awesome chip. About £8k for a DevKit though.

Yeah, I think DRAM is almost certainly the future, just in terms of being able to afford the memory capacity to fit large models. Even Cerebras using a full wafer only gets up to 44 GB of SRAM on a chip (at a cost over $2M).

An interesting twist is that this DRAM might not need to be a central pool where bandwidth must be shared globally -- e.g. the Tensortorrent strategy seems to be aiming for using smaller chips that each have their own memory. Splitting up memory should yield very high aggregate bandwidth even with slower DRAM, which is great as long as they can figure out the cross-chip data flow to avoid networking bottlenecks

Reply on news.ycombinator.com