The other element which appears unsaid is that in a typical datacentre, your bisection bandwidth is typically << (num computers * network bandwidth per computer). Or in other words, even if your computation is bandwidth and not latency starved, you're not obviously going to be able to do a gigantic data shuffle quickly unless it's within a rack. This is to say, I'm not sure _just how much_ this would affect your overall target system architecture at present. Once you're switching at a couple of terabits things are likely quite different in those terms.

The other element which is a bit scary is that it's fairly rare these days for mass market companies to index deeply on tech which isn't available in public clouds, so until AWS supports this sort of thing, it's unlikely that many folks will target it. You can kind of see this with the Optane PDIMMs - they looked absolutely fantastic, but given you couldn't get them on any AWS instance there wasn't much point actually trying to use them outside of very specific applications - as a software engineer this hardware lets me build my software very differently and in a simpler way, but how can I possibly risk architecting based on that if it then cannot support a customer's cloud migration?

Obviously v different in HPC contexts.

And it should always be said - latency is very different between these. Memory latency still measured in nanoseconds, PCIe latency still measured in 10s of microseconds, about 3 orders of magnitude difference.