Why Fugaku, Japan's fastest supercomputer, went virtual on AWS

its not a fallacy, its a tradeoff. If you want to have 100nodes doing batch work in a datacentre, you need to buy, install, commission and monitor hardware, storage and networking.

Now, That is possible, but hard to do on your own. So realistically you'd get a managed service to install that stuff in a data centre. As your experience will show you, the level of service you get is "patchy". You really need some one who has worked on real steel to specify that kind of work. Those people are incredibly rare, especially if you want someone who can talk the rest of the stack as well. (That is one massive downside of the cloud, there are hardly any new versatile sysadmins being created.)

As a former VFX sysadmin who looked after a number of large HPC clusters, its fairly simple to get something working, but something running reliably and fast is another matter.

> Most people vastly overestimate the scale they need

Yes, but we are talking about an HPC cluster that is >100k nodes. Something fast enough to cause hilarious QoS issues on most storage systems. especially single rack arrays with undersized metadata controllers. Even on isilon with its "scale out" clustering, one job can easily nail the entire cluster by doing too many metadata operations. (protip single namespace clusters are rarely worth the effort outside of a strict set of problems. Its much better to have a bunch of smaller fileservers grouped together in an automount. Think of it like DB sharding, but with folders.)

Second tier cloud providers are rarely able to provide 10k GPUs on the spot, certainly not at a reasonable price. Most don't have the concept of spot pricing, so you're deep into contract negotiations.

> AWS support didn't leave me in awe.

Its not great unless you pay. but you will eventually get to someone who can answer your question. Azure on the other hand, less so.