China Tops U.S. in Supercomputers

Which just emphasizes the point that most applications would be better served by cheaper, more resilient hardware.

It's true that we need shared data, but we don't need shared state (memory) for any of our workloads, and shared data (e.g. disk/db) is a much easier problem to solve than shared state.

There is still coordination and orchestration, but that's an extremely coarse amount of communication compared to the cost of Cray interconnects. That being said, there are cases where we might benefit from the Cray networking, but that comes at the cost of other tradeoffs (no local disk, low memory per core).

So what do we do? Well, we use a handful of Ferraris to get the job done because they are available when cheap bus would have suited us fine. The double whammy is that the Ferraris end up in the shop all the time and occasionally somebody else gets exclusivity to them when they want to get in a Bell prize submission.

On the other hand, you can just run N independent simulations, and pool the statistics. In many cases, this is as good as if not better than running one long MD simulation. Here, all the processors share a common input file, but it's a single file, it only has to be communicated once. Any results from the processors are compacted, and sent to a large file server, which in itself is designed with partitioning since all the output files are independent. This approach won't let you work with large systems (unless your individual servers have more RAM), but it scales linearly to far more processors than tightly coupled parallelism, for far cheaper, and often, the scientific results are as good or better.

This shows up in many cases- in my 20+ year career programming, nearly all the problems I saw people run on supercomputers eventually could be run on CPUs with easier code and better performance. There are some irreducible codes for which only supercomputers are capable of meeting the required specs, and those are the only codes which should run on supercomputers. Otherwise you're pissing money on interconnect and hard to program environments).

In short: if you can convert your supercomputer job into a cloud cluster, you should do so, because it will save time and money. If you can't, then you should use a supercomputer.

Note also that cloud interconnects have gotten significantly better over the years- for me, 100Gbit throughput and latency (as long as the cluster has the total parallel bisection bandwidth) is fine for me and most other people.

The interconnect on the Sunway TaihuLight seems to be standard Infiniband, from reading other articles about the system.

So no, these machines are often used for problems which really do utilize the large-scale nature of the computers.

So lots of the "big data" type problems with lots of uncoordinated jobs aren't going to take advantage of the primary component of these big computers, the node interconnect. The computers really shine with big fluid dynamics simulations where there's lots of chatter between different components of the simulation to communicate boundary conditions and the like.