Specifically, a PCIe 3.0 x16 slot can't even support a 200 Gbps NIC. Apparently the Mellanox ConnectX-6 NIC takes up two slots.
What's interesting in this article is that the bus is becoming a local bottleneck between various system resources. (cpu/gpu/accelerators)
Even having enough bandwidth, handling 25 GB/s data, except for hardware-accelerated network processing, is very hard when current processors have 60 GB/s of RAM bandwidth (DDR4, dual channel), and even with zero-copy modern OS is going to be hard for the CPU to process the packets on continuous full load. That's mind blowing, and amazing :-)
And yes, IB is designed such that the NIC HW can offload quite a lot, and the rest is indeed done in userspace without kernel involvement in the hot paths.
That being said, it's possible to run more or less the IB protocol stack on ethernet hardware, it's called ROCE (RDMA over converged ethernet). Somewhat amazingly, latency is actually quite competitive with IB.