One interesting thing is if the model can’t fit into GPU memory (sharded across multiple chips) it would be much slower. So one chip would take more than a month.

That’s why these clusters are so valuable, even with Nvidias margins they are still cheaper than using less compute for longer.

Could it actually? Or do you need the memory size of thousands of them running in parallel?