Depends on your bisection bandwidth requirement and software cost. If you want to do MPI it is going to be cheaper to buy some InfiniBand cards. But if you have a small cluster and tightly coupled code, or large I/O requirements, then a PCIe switch might be the go. These get used on some video on demand systems and signal processing architectures:

https://www.microsemi.com/products/drivers-interfaces-and-pc...

The Dolphin systems are tuned better for computation: http://www.dolphinics.com/products/IXS600.html

I expect small PCIe NVMe external storage systems to become quite common in the near future, because enterprise systems need multipath storage for reliability; 8GB/s FC is too slow for SSDs, let alone NVMe, same with SAS bus expanders.

https://events.linuxfoundation.org/sites/events/files/slides...