"Transformers are memory bandwidth bound" - this is the precise reason why this makes sense. If a model doesn't fit into memory on a single device, it needs to be incrementally loaded into memory (offloading), which is bottlenecked by memory bandwidth. Splitting the model over multiple devices avoids this, instead trading off for latency of communicating between nodes. The network bandwidth requirements are minimal since only the activations (intermediary embeddings) are passed between devices. For Llama-3-8B these are ~10KB, for Llama-3-70B these are ~32KB.