GPU memory bandwidth is the limiting factor, not PCIe bandwidth. The memory bandwidth is critical because the models rely on getting all the parameters from memory to do computation, and there is a low amount of computation per parameter, so memory tends to be the bottleneck.