Rule of thumb is that you need ~20 tokens per parameter. The average token size is ~4 characters, probably more for larger models where you want larger dictionary, but for simplicity I'll say it's 5 bytes to make numbers round. So you need 100 bytes of text data per parameter, or 10 PB for 100T model. Now, recent research says that you can reuse the same data like 4 times before it becomes hindering performance but it doesn't help much in our case.

But in this case what is really ridiculous is the compute requirement. The required compute for optimal model growth roughly quadratically (both your model and your data grow linearly). So for 100T model you need 1e30 FLOPs. This machine gives you 1e18 FLOPs per second. It will take 30k years to train this model on one of these (or 30k of these to train it in a year, but then utilization will start kicking in).