We connect hundreds of chips across several racks with fast interconnect.
How fast is the memory bandwidth of that fast interconnect?
Have a look at section 2.3 of our paper. Between any two chips we get 100 Gbps. The overall bandwidth depends on the connection topology used. I don't know if we make that public.