Working with Mellanox cards for a few weeks; the MTU matters a lot. The difference between an iperf test measuring 14 Gbits/s on MTU 1500 vs. 40 Gbits/s on MTU 9000.
We got a couple of Connect-X5 cards, which allow switchless connections, akin to a ring topology. A lot of neat things, at just stupid line speeds, and latency levels I haven't seen in software, ever.