Hacker News Reader

Comment by melling | Hacker News Reader

Training can take hours, days, weeks, or months.

Doesn’t the inference part only require seconds? Since it requires a fraction of the computation , can’t CPU’s be optimized for that? A few matrix multiplications

View on HN

Training an LLM can be batched - you can train using entire sentences / blocks at a time. But when doing inference, you need to do one word at a time so you can put the output word back into the input.

The optimization problem is that it’s often not the CPU that’s bottlenecked. It’s RAM. As I understand it, if you run llama locally you need to matrix multiply a few gigabytes of input data for every output token before your computer can start figuring out the next token. Since the weights don’t fit in your CPU’s cache, DDR bandwidth is the limiting factor, just pulling all the weights over and over into your cpu. GPUs are faster in part because they have much faster memory busses.

To really optimize this stuff on the cpu, we need more than a few new CPU instructions. We need to dramatically increase ram bandwidth. The best way to do that is probably bringing ram closer to the cpu, like in Apple’s M1/2 chips and nvidia’s new H100 chips. This will require a rethink of how PCs are currently built.

Inference is only seconds on a GPU, but have a look at flops of modern GPUs to CPUs - matrix multiplications differ by two orders of magnitude. Seconds on the GPU is minutes on the CPU. And don’t forget inference needs to scale in the data center, it needs to run repeatedly for many users.

It could, and they are. But that's only relevant if you're running the model locally. If the model is being ran at scale, then throughput matters and GPU's would be king still.

GPUs will be better optimized for large matrix multiplications than CPUs, by design.

And you need to the inference again and again, not just a single time (like your training).

Desktop CPUs have integrated GPUs, so it's more complicated. If I infer on the GPU inside my CPU, how do you count that?

True. Memory bandwidth may be the most limiting factor.

Reply on news.ycombinator.com