Hacker News Reader

RAM is (often) the bottleneck for highly parallel GPUs, but not for CPUs.

Though the distinction between the two categories is blurring.

Memory bandwidth is the bottleneck for both when running GEMV, which is the main operation used by token generation in inference. It has always been this way.

Reply on news.ycombinator.com