I think Meta’s cluster will be slightly larger than 100k next month, scaling to 350k by end of year. And double of that sometime next year. According to their info.
I haven’t seen any info on those being in a single cluster. They seem to just be referring to total GPUs. It’s currently almost impossible to create a single cluster of GPUs that size.
Ah correct! It seemed weird to me they would have that figured out while others have not.