Hacker News Reader

I have a naive question. When people talk about needing x amount of vram are they meaning all on one card, or can it be distributed across multiple cards?

Assuming power consumption is not an issue, Can these loads be effectively run on farms of systems using older cards with smaller vram sizes?

How about for training? Are the hardware requirements fundamentally different apart from scale?

View on HN

Model parallelism is possible for inference and you can host extremely large models across multiple accelerators (see DeepSpeed ZeRO), but the inference speed is several times slower compared to just having all of it on one accelerator due to communication bottlenecks and overhead (parallelism refers to memory needed for parameters, not the actual computation in most cases).

Certain models may require storage formats like bfloat16 to run efficiently that may not be supported on older hardware.

Parallelism is also supported (and more necessary actually) for training, but since backprop is expensive it typically requires many multiples of memory requirements needed for inference.

Reply on news.ycombinator.com