96-core ARM supercomputer using the NanoPi-Fire3

Had to read your answer twice... "High latency???" But I see you consider 1-2us high latency. :P

It is an interesting question what people who don't have access to a supercomputer, but would like to learn and optimize for HPC-style distributed memory programming should use.

I've found AWS to be pretty nice, except there are no RDMA drivers for the elastic NIC and the BW is a bit low. (25Gbit vs. 100Gbit). For MPI bulk synchronous programs, it's probably a pretty close model, though.

> No. Containers on a single node are too fast.

You can add artificial delay to your local container network to better simulate a production environment.

For example using https://github.com/alexei-led/pumba and "pumba netem delay" you can add networking delay between Docker containers, and "pumba netem rate" can limit the bandwidth between them as well.

("pumba" is just using the underlying Linux networking traffic control technologies, such as the "tc" command from "iproute2", so you don't have to use "pumba", you can set this up manually, but a tool like "pumba" makes it a lot easier.)

> handle communications within dozens of nanoseconds. So its a bad "emulator" for SuperComputers

But ping on local host reports .3 ms latency..?

In any case it’s still an easy way to get started, and arguably when exact latency starts counting for something, you have to be tuning your code on the system it’s going to run on. An RPi cluster could skew that in all sorts of ways, eg the TCP stack being disproportionately slower etc

This part:

> how your massively parallel algorithm behaves on a 96-node cluster / network, such a box is just $500, and is portable and can work offline.

That page is embarrassingly wrong about how power management works in Intel CPUs. By default Intel CPUs will not allow their rolling average power consumption over a period of ~1 minute to exceed the specified TDP (95 W in this case). Once the limit is reached the CPU reduces its frequency to bring power consumption down. Intel optimizes their CPUs to achieve a good balance between efficiency and performance when operating at the TDP.

What you see in Anandtech's review is the result of motherboard firmware effectively disabling the power limit by setting it to a very high value. This is a common practice among enthusiast motherboards in order to boost scores in reviews. Unfortunately it also results in drastically lower power efficiency and lots of clueless people, including many tech writers, complaining about unrealistic TDP numbers.