This is pretty awesome. Does anyone know if google or azure can scale to 40,000 cores on demand like that?
I work at DL (although not technical team) and can attest that this is attainable for GCP, and we do so on a fairly regular basis. I cannot speak to Azure, but I expect it too is likely.
I'm building a 160,000 core cluster on GCP right now. It took a week to get the quota increase approved, but GCP can definitely handle it.
Would love to see another post where this is run on Google and Azure as well and the results are compared.
what's the cost on something like this?
It cost less than $5000 to run on AWS. More if you include engineering's time.
per month? pretty amazing
No, just $5k total for this test. Would be much more if this was run for a month straight.