From the article -- “The true bottleneck for processor throughput is the network interface used to connect a machine to an external network, such as an Ethernet, therefore severely limiting a processor’s data ingest capability,” said Dr. Jonathan Smith, a program manager in DARPA’s Information Innovation Office (I2O). “Today, network throughput on state-of-the-art technology is about 1014 bits per second (bps) and data is processed in aggregate at about 1014 bps. Current stacks deliver only about 1010 to 1011 bps application throughputs.”

I don't disagree with this, but see the challenge not so much as the transmission speed between nodes, rather I see it as the 'semantics' for expressing addressing and placement. (and yes, I am a big fan of RDMA :-))

One of the things I helped invent/design back when I was at NetApp was some network attached memory or NAM. NetApp was among a number of people who realized that as memory sizes got larger, having them associated with a single node made less and less sense from a computational perspective.

One can imagine a "node" which lives in a 64 bit address space, has say 16GB of "L3" cache shared among anywhere from 12 to 12,000 instruction execution or 'compute' elements. More general purpose compute would look more like a processor of today, more specialized compute looks like a tensor element or a GPU element.

"RAM" or generally accessible memory would reside in its own unit and could be 'volatile' or 'involatile' (backed by some form of storage). With attributes like access time, volatility, redundancy, etc.

That sort of architecture starts to look more like a super computer (albeit a non-uniform one) than your typical computer. With nodes "booting" by going through a self-test and then mapping their memory from the fabric and restarting from where they left off.

I thought I might get a chance to build that system at Google but realized that for the first time in my career that I was aware of I was prevented by my lack of a PhD. And what was worse, given the politics at the time, no one with a PhD would sign on as a 'figurehead' (I reached out to a couple and we had some really great conversations around this) because of the way in which Google was evaluating performance at the time. (think grad school ++ but where only one author seems to get all the credit).

Now that I've had some free time I've built some of it in simulation to explore the trade-offs and the ability to predict performance based on the various channels. Slow going though.