Hacker News Reader

Comment by bogwog | Hacker News Reader

So NVLink/NVSwitch pools multiple GPU resources on a single (very expensive) system. A cheaper alternative to that is "offloading", which is a technique that splits the inference process into smaller steps, so it can run on systems with much less resources available... and Petals is a 10x faster alternative to that.

Did I get that right?

This AI stuff is moving very fast and it's hard to keep up, but it's all fascinating.

View on HN

You're right. This comment explains offloading in more detail: https://news.ycombinator.com/item?id=34216213

Offloading is when the computation is done on the CPU instead of the GPU. DeepSpeed is an example of this.

In case of offloading, the computations are usually still performed on GPU, but the model is hosted in RAM/SSD instead of the GPU memory (and its chunks are copied to the GPU memory when necessary).

A lot of computation is offloaded to the CPU, such as gradients and optimizer states. You are right though that quite a bit of computation is still done on the GPU.

I remember when GPUs were starting to support arbitrary computation and offloading meant shifting work away from the CPU.

Matches my understanding also. Can someone in the know confirm?

Your understanding is correct, but I can't vouch for the claim's accuracy. This could make the execution of models much more accessible to people who don't have a 4x / RTX3090 or better in a ML or mining rig ..

Reply on news.ycombinator.com