Facebook's

> When RSC is complete, the InfiniBand network fabric will connect 16,000 GPUs as endpoints

Microsoft's

> The supercomputer developed for OpenAI is a single system with more than 285,000 CPU cores, 10,000 GPUs and 400 gigabits per second of network connectivity for each GPU server