We have a few clusters at work that we use for numerical weather forecasting using the WRF weather model and some other custom applications. And, indeed, the InfiniBand interconnect was a hefty portion of the overall cost of the clusters. At one point there were people considering Gigabit Ethernet instead of IB, but thankfully it was shot down. The lower latency with IB has a greater impact on MPI-based applications than some people realize.