One also desperately wishes that multi-host network adapters had a low-end market. A 1 x 25Gbe connection shared between 6 hosts would be epic. Or if sharing the connection is too much, just a 4 x 2.5Gbe NIC chip would be acceptably interesting & useful to have. I forget, maybe the gbe is free on the Pi4's chip & this is more of a non-issue.
Hardware: these PCIe switches are finicky as all get out; will it link up to your device reliably? Who knows, depends on how the switch and device manufacturers interpreted the PCIe specs.
Software: there is no nice easy standard for connecting host-to-host (which I’m assuming is what one would want on a multinode system Like the one linked). There exists the idea of a PCIe non-transparent bridge, which if I recall correctly, allows CPUs on either side of it to do blind memory reads and writes to each other (hope your IOMMU is set up correctly to protect you from ugliness that could occur here).
Long story short there’s a reason PCIe is a local bus; it was never designed to do anything else. Anything else is largely a proprietary bolt on.
In addition to mapped memory, there are some other modest capabilities: support for ringing the peer's doorbell (one way), & scratchpad configuration. There's a pretty good 2016 presentation on NTB in Linux here: https://events.static.linuxfound.org/sites/events/files/slid...
Interesting to note, Nvidia has docs for settings up NTB on their Pegasus & Xavier systems: https://docs.nvidia.com/drive/drive_os_5.1.6.1L/nvvib_docs/i...
I for one think more people should be investigating & harnessing the open-source Linux NTB tools. Switches are a bit expensive, but compared to ethernet switches their cost/throughput is great. Alas, yes, the whole ecosystem is woefully undersupported, under-developed. Isn't PCI part of the whole PC thing, this great revolution in computing that allowed interoperability to rise? Such a pity that all the PCI SIG's seemed to fizzle out, that not a lot of people participated. Sure we got some SR-IOV but that feels like as far as we got.